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1 EXECUTIVE SUMMARY 


The goal of this project is to conduct research leading to identification and 
analysis of predictive modeling capabilities to support behavior influence 
operations. The key deliverable is a set of recommendations for mature 
analytical tools that meet the needs of National Air and Space Intelligence Center 
(NASIC) Behavioral Influences Analysis Division leaders and analysts. Though 
text mining and social network analysis tools are important for this customer, the 
primary focus of this effort was to identify the best Bayesian modeling capabilities 
W this mission area. 

Project work was executed by a govemment/contractor team of software, 
intelligence, modeling, cognitive science, operations research, engineering, and 
psychology experts from NASIC, AFRL, and SRA International, Inc. Project work 
started in August 2003 and completed in December 2003. 

SRA personnel executed this project by dividing the work into four tasks: 

• Task 1: Develop capability-based criteria 

• Task 2: Market research - identify simulations, models, tools 

• Task 3: Rigorous, objective assessment of simulations, models, tools 

• Task 4: Develop recommendations regarding acquisition of tools 

Detailed Criteria, developed from a thorough understanding of the mission 
requirements, fell into four criteria categories: General Features. Input 
Manipulation, Performance Extensibility, and Interoperability. Using contacts 
identified by the customer, our own knowledge of the market, and additional 
investigations using keyword searches of relevant databases and the worid-wide- 
web, the team developed an extensive list of over 80 candidate tools. From this 
list, we selected a prioritized set for further data collection and detailed analysis. 

The Analytical Hierarchy Process (AHP) technique was applied to evaluate 
and rank the tools. From a performance standpoint, Hugin ranks as the optimal 
Bayesian tool among the alternatives. The next best performance alternatives 
are BayesiaLab and Netica. From an “ease of use” perspective, Netica and 
Analytica rank highest. Cost and pricing structures are roughly comparable, with 
the notable exception of SIAM, which is available free of licensing fees to 
government organizations. 

Our conclusion, based on this analysis, is that several mature alternatives 
are available to NASIC. Investment decisions could be made on the basis of this 
report, but a more sound approach would be to develop and run a benchmark 
problem scenario on a demonstration version of the highly rated alternatives. 


1 







2 BACKGROUND 


2.1 Project Description 

The goal of this project is to conduct research leading to the identification 
and analysis of predictive modeling and analysis tools and capabilities to support 
behavior influence operations. The key deliverable is a set of recommendations 
for mature analytical tools that meet the needs of National Air and Space 
Intelligence Center (NASIC) Behavior Influence Analysis Division leaders and 
analysts. 

Project work was executed by a multi-disciplinary, government/contractor 
team of software, intelligence, modeling, cognitive science, operations research, 
engineering, and psychology experts from NASIC, AFRL, and SRA International, 
Inc. Project work started in August 2003 and completed in December 2003. 

The customer for this work is Greg Jannarone, Chief of the relatively new 
Behavioral Influences Analysis Division at NASIC (NASIC/BPB). Gilbert G. 
Kuperman, Principal Mathematician in the AF Research Laboratory’s Human 
Effectiveness Directorate, and members of his staff also participated in this effort. 

Government personnel provided some valuable early guidance and 
organizational contacts to get us started in this effort. They provided a basic 
cultural/institutional/psychological framework for the end-state model, desired 
capabilities that helped us articulate requirements, thoughts on initial evaluation 
criteria (e.g., easy to use, cheap, can access all information/data sources 
needed, accurate, reliable, technically mature, risks are manageable, 
supports/enhances key capabilities, extensible, flexible, interoperable or easy to 
confederate, supportable, etc.), and finally some valuable leads on people and 
organizations who could help us get started. 

This project is envisioned as a foundation for future work by NASIC analysts. 
They will use the tools identified in this study to assemble data and build models 
of individuals and small groups of interest, ultimately leading to the identification 
and exploration of predictive modeling, simulation, and analysis capabilities in 
support of USAF (and potentially Joint) behavioral influence operations. 

SRA personnel executed this project by dividing the work into tasks (listed 
below and described in more detail elsewhere in this report): 

• Task 1: Develop capability-based criteria to assess predictive 
constructive simulation, descriptive models, other analytical tools 
(review NASIC goals for products, end states, analytical processes) 

• Task 2: Market research - identify simulations, models, tools 

• Task 3: Rigorous, objective assessment of simulations, models, tools 
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• Task 4: Develop recommendations, regarding acquisition of tools 

2.2 Architecture Context for This Work 

AFDD 2-5 [10], the USAF’s Doctrine Document on Information Operations 
states that “Influence Operations are an integral part of modem aerospace 
strategy,” and defines Influence Operations as one of three core Information 
Operations capabilities (the others are Electronic Warfare Operations and 
Network Warfare Operations). According to AFDD 2-5 (2003 draft): 

Influence operations are focused on affecting the perceptions 
and behaviors of people, leaders, groups or entire populations. The 
means of influencing can be physical or informational. The 
cognitive domain is composed of separate minds and personalities 
and influenced by societal norms. The cognitive domain isn't 
homogenous, continuous or even necessarily rational. 

2003 Draft AFDD 2-5, Chapter 1, Page 3 

According to the CONOR for AF Effects Based Operations [11]: “The most 
significant challenge is to link (trace, understand, predict, assess) actions 
[alternatives executed by blue forces] to [adversary] behavioral outcomes.” 

The 29 July 2003 draft AF Psychological Effects-Based Operations (AF 
PEBO) Architecture OV-2 (Paragraph 2.4.5.2) [12] defines one important node in 
the architecture as a NASIC Behavior Influences Analysis Branch chartered to 
“provide relevant intelligence analysis and products supporting AF PEBO and 
joint PSYOP targeting and planning.” The stated mission of this organization is 
to “conduct human target vulnerability analysis.” An SRA analysis of the trace of 
this NASIC mission to other elements of the overall Influence Operations and 
PEBO Architecture is provided as Attachment 1. 

Modern intelligence/operations architectures will ultimately deliver increased 
capability to the warfighter, but only if the architecture enables true integrating 
strategies among relevant functional experts. Pre-operation estimates of 
adversary reactions to influences are certainly possible. Individual and small 
group behavior is rooted in culture, organizational/institutional ties, and unique, 
“knowable” psychological factors. A holistic, fused, all-source approach is 
common to all successful operations, and is critical to connect cultural, 
institutional, and psychological influence factors. However, the potential universe 
of information and possible relationships to characterize human targets of 
interest is very large. Therefore, analysts and decision makers need tools to 
assist them with this mission. 

Ultimate customers for the capabilities developed from follow-on efforts to 
this project will include PSYOP, Deception, HUMINT, Information Operations, 
Kinetic Operations, and PEBO planners/targeteers. 
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2.3 Project Definition and Scope 

2.3.1 Objectives 

Current Project Product Objectives: 

• Recommendations regarding acquisition of COTS/GOTS tool suite, 
based on rigorous, objective assessment of existing tools 

• Include performance, cost, and usability criteria 

Customer Process Objectives Supported Bv This Project: 

• Support accurate probabilistic prediction of behaviors in response to 
input COAs 

• Enhance NASIC effort to support effects-based operations 

• Determine key psycho/social factors that influence targeted individuals, 
groups 

• Develop templates for relative influence weights/probabilities of 
influence 

• Develop estimative confidence levels for decision-making factors 

• Estimation process for “predicting” potential anomalous/irrational 
behavior 

• Probabilistic model of likely decision-making behavior 

2.3.2 Project Framework 

SRA planners felt it was necessary to fully understand and appreciate the 
elements of the capability envisioned to meet the needs of our customer. One 
way to capture this understanding is by building some alternative frameworks for 
the capabilities. Figure 1 shows two alternative views of an implementing 
framework for the NASIC vision. 
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Figure 1. Two alternative Project Capability Frameworks 

What are we trying to represent within either of these frameworks? First, 
there is a need to characterize target (in this case individual human beings or 
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groups of human beings) vulnerabilities: what target characteristics or attributes 
can planners and analysts leverage and exploit? In conjunction with this, there is 
a related but distinct need to characterize target susceptibilities: can the target 
behave in the way we expect or desire? Is there a predictable path to exploit the 
vulnerability? Finally, there is the need to determine accessibility: can our target 
of interest be reached (physically or other) to use the susceptibility and exploit 
the vulnerability? 

Within the layers/filters of the frameworks, note there are categories of target 
attributes that may be exploitable: 

• Psychological attributes - What can we know (emotions, ambitions, 
motivations, goals, needs)? 

• Institutional attributes - What happens when we sever or influence key 
links? 

• Cultural attributes - Which of the numerous candidate cultural 
attributes do we really care about? Which are the best contributors to 
a prediction of target behavior? 

What we’re really looking for in an influence operations framework, then, are 
influence links that originate in culture, pass through organizations and 
institutions, into the minds of individuals and groups - thread(s) that reliably link 
the three planes of influence can put us on the path of a predictive capability. 

Any selected tool or suite of tools must provide the environment for implementing 
this capability. 

SRA’s investigation identified three categories of tools that are needed to 
implement the analysis framework alternatives illustrated in Figure 1. These 
categories are: 

• Text/Data Mining capabilities 

• Social Network Analysis capabilities 

• Modeling capabilities 

Though this study took all three categories into account, most (90%) of the 
focus for the SRA team’s effort was on identifying and analyzing modeling 
capabilities. This decision was made, with the full support of the customer, to 
ensure the main project objectives could be achieved within resource constraints. 
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3 REQUIREMENTS 


3.1 Vision 

NASIC Behavioral Influences Analysis Division Vision : Identify influences that 
can condition, alter perceptions, get the attention of and (ultimately) effect the 
t>e/ 7 awor of adversaries (individuals, small decision-making groups). 

3.2 Mission 

Division Mission : Help the AF build a concept of operations that integrates 
intelligence and information operations/warfare communities as an “influence 
operations capability/system” {that links actions and behaviors). 

3.3 Critical Immediate Need 

Critical immediate need : Identify modeling, simulation, and analysis tools and 
capabilities for pre-operation estimation of behavioral influence operation option 
effects. Need is urgent for tools/capabilities to help analysts establish a solid 
analytical foundation for this emerging mission area. 

3.4 NASIC Analyst Input 

On 19 Sep 03, an SRA team interviewed four NASIC/BPB Branch Chiefs 
and the BPB Functional Team Lead [13]. The following paragraphs summarize 
our interviews. This is presented because it helped form a critical link between 
high level requirements presented in the PEBO Architecture (see Paragraph 2.2) 
and the criteria we developed for this study. Analyst questions are provided as 
Attachment 2. 

SRA’s main goal was to capture the analysts’ focus, expectations, and 
priorities as it pertains to the current project. Following are some top level 
observations: 

• Analysts with different areas of responsibility will have some 
overlapping focus areas, needs, expectations, and priorities, but will 
also have unique needs associated with unique problems in their 
assigned area. 

• Some will focus on small groups and individuals in the near term. 

• Some will rely more on organizational and cultural information because 
detailed information on individuals will be hard to come by. 

• Analyst skill set will range from relatively new analysts to 20-year 
analysis veterans with knowledge and expertise in multiple “INTs.” 

• Analysts will support a range of customers (intelligence, planners, 
operators, targeteers, IWF personnel), in deliberate planning and crisis 
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action modes, primarily at the operations and strategic levels of 
conflict. 

• Typical products will include target profiles, target folder supplements, 
target system analyses, short-suspense RFIs, pre-positioned 0-Plan 
products, and vulnerability assessments. 

• Available information should be the LIMFAC, not the analytic tool set. 

• Analysts are very concerned about data overload and data 
organization, not as much about target models at this point in time. 
Their priorities focused on user-friendly, data mining (associative 
pattern mining) and relational database tools first, model building and 
predictive tools second. 

• Analysts do not expect an analytic tool set to replace their knowledge 
and expertise. They expect that they and their functional experts will 
provide the knowledge needed for the modeling capability. Aids to 
help remember and organize knowledge would be welcome capability 
additions. 

• Models will need to handle at least 300 nodes, perhaps up to 3000 
nodes. 

It became evident after these interviews that the SRA team needed to focus 
on three technology classes to provide the overall analytic capability needed by 
BPB analysts: text/data mining tools (to group and summarize documents), link 
analysis tools (e.g., relational database mining tools to find links between 
individuals and groups/events/attributes), and modeling tools (to build belief 
nets). Some of the specific needs/desires for an analytical capability articulated 
by the analysts include: 

• Flexibility to adapt to needs of individual analysts 

• Ability to grow in size/sophistication as information set matures 

• Ability to search by time relationships, group associations, family 
relationships, geographic proximity, etc. 

• Access classified (multi-INT) and open source information 

• Efficient pulling and loading (text, photos, e-files, paper, web pages, 
etc.) 

• Capability to provide reminders or triggers on saved data 

• Efficient organization of information so it can be easily found when 
needed 

• Liked the SIAM logic train (how it maintains track to source) 

• Want tools to help them “lay the evidence out” 
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• Capability to assign probabilities of given events (not necessarily to 
have modeling tool assign the probability) 

• Spider diagrams will aid social network analysis 

• Want interoperability with other agencies’ tools (e.g., Visual Links, 
Analysts Notebook) 

• Willing to be trained, but the less training needed, the better 

• Data/text mining and relational data capabilities will populate a 
hierarchy of cultural, organizational, individual events 

• Want node/link structure, weights easily tailored to a given target 

• Analysts have techniques to search for (and acquire) what they “know 
they don’t know;” analysts would love to have aids to help them with 
what they “don’t know they don’t know” 

• Volume of information can be overwhelming to an analyst; they need 
help to sort, organize and prioritize - 1300 messages per day 
(classified/unclassified sources) and improve workload efficiency 

• Understanding bias associated with sources is a very important 
element of the foundation of a predictive capability 

• Text is important to everyone, visual information (e.g., pictures) is 
critical for some due to the shortage of useable text information 

It also became evident that there are some things the analysts will not be 
expecting from an initial analytical tool set: 

• Will not be likely to trust search/data mining tools to find all relevant 
relationships - analyst will still play a major role in reviewing, 
organizing, and making inferences from the available data 

• Multi-media will be useful, but is not a priority now 

Consistent priorities for the analysts are, in this order, user friendly 
sort/organization aids, relational database, links/nodes model structure, Bayesian 
nets for probabilistic assessments. 

These interviews were very valuable to the SRA team. This information 
reinforced and clarified the user requirements, and facilitated our completion of 
project Task 1. 

3.5 Final Project Requirements Trace 

Attachment 1 presents a final requirements trace from Influence Operations 
doctrine and architecture (requirements) to the criteria we used to evaluate 
candidate tools and capabilities. This trace ensures that the criteria used to 
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evaluate promising tools are measuring attributes that contribute to the 
accomplishment of the vision, mission, and requirements of NASIC/BPB. 

4 TASK EXECUTION 


4.1 Methodology 

The basic methodology used to execute this project is described in the 
following steps: 

• Document/present a project plan 

• Ensure requirements are well understood 

• Map requirements to criteria (emphasis on modeling tools) 

• Establish a list of candidate tools by conducting a market survey 

• Develop questionnaire tied to the evaluation criteria 

• Phone or personal contact with vendors, followed up with a request to 
complete the questionnaire 

• Enter data from completed questionnaires (follow up as necessary) 
into a spreadsheet format 

• Analyze data 

• Report results to customer 

The project methodology was executed in phases as described below: 

• Phase 1: Detailed planning 

• Review available guidance from customer 

• Contact people who understand mission requirements 

• Develop information collection strategy & methodology 

• Assign initial team roles, responsibilities 

• Build a detailed project schedule 

• Develop an initial capability and criteria list 

• Present project plan to customer for feedback 

• Phase 2 (Tasks 1 and 2): Refine capabilities/criteria, complete market 
research 

• Identify project/product needs, “alterables”, constraints 

• Define a product framework 

• Identify functions that are needed within the framework 
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• Identify technology categories needed to implement the functions 

• Develop criteria to evaluate tools, technologies 

• Effects-based criteria (ability to do the job) 

• Other criteria (ease of use, cost, supportability, etc.) 

• Phase 3 (Tasks 3 and 4): Collect data, perform assessment on a 
prioritized subset of the identified tools 

• Identify existing tools that implement required technologies 

• Evaluate tools against approved criteria 

• Phase 4: Final report writing, recommendations 

• Recommend right mix of tools 

• Immediate capability expectations 

4.2 TASK 1: Develop Criteria 

Task: Develop capability-based criteria used to assess “predictive constructive 
simulation, descriptive models, and other analytical tools” (review NASIC goals 
for products, end states, analytical processes). 

Task 1 Deliverable : Spreadsheet detailing and organizing the project evaluation 
criteria (Attachment 3). 

Using the Requirements Trace described in Section 3 and presented in 
Attachment 1, the team outlined a basic structure of needs (important 
characteristics that are “must-haves” for the project), “alterables” (criteria that 
could be compromised and adjusted to fit the structure of the project), and 
constraints (attributes that would limit project scope). These needs, alterables, 
and constraints are presented below: 

Needs 

• Supports target audience development 

• Vulnerabilities, susceptibilities, protections 

• Supports identification of a range of influence factors 

• Can assign weights to influence factors (culture, institution, 
psychological) 

• Connects influence factors with candidate audience behaviors 
(links/nodes) 

• Relational database handling/management/processing 

• Can access all data sources needed (U through TS) 
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• SIPRNET-compatible 

• Reduces analyst workload 

“Alterables” 

• Easy to use (minimize training requirements) 

• Low cost 

• Accurate 

• Reliable 

• Mature 

• Risk is manageable 

• Extensible (growth capacity) 

• Flexible 

• Interoperable or easy to confederate 

• Supportable 

• Source code accessible or stable API 

Constraints 

• Bandwidth (text, relational database inputs) 

• Low level of contractor support needed 

• Don’t want to use as stand-alone tool (open arch, common formats) 
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Comparing the needs, alterables, and constraints with the model framework 
described previously, it became clear that certain functions would be required to 
satisfy the basic design of an Influence Operations analytical capability. These 
functions are illustrated in Figure 2. 

Figure 2 describes three major functions: input, manipulation, and access. 
Within these functions are sub-functions (e.g., ‘publish, subscribe, transform, 
query,’ etc.). Implementing these sub-functions will require specific technologies. 
Identifying and listing these technologies contributes to another level of 
understanding with which to build evaluation criteria. An initial cut (developed in 
the early phases of the criteria development phase) on the specific technologies 
needed to implement Influence Operations framework functions is presented 
below: 

INPUT Technologies; 

• Identification & authentication: Source certificates. Secure ID, Source 
availability. Source discovery, Source ID 

• Access and translation: Multimedia data capture. Wrapper technology. 
Heterogeneous data integration. Transformation techniques. Capture 
plan data. Meeting transcription 

• Upstream information: Tagging techniques, Source characterization, 
Capturing user intent. Pedigree capture (source processing) 

• Categorization: Task/data relevance. Ontologies & taxonomies. 
Expectation driven change detection 

MANIPULATION Technologies: 

• Storage: Multimedia storage. Resource distribution management. 
Multilevel secure storage. Seamless access to tertiary storage. High- 
performance computing. Backup and recovery. High-density mass 
storage. Data warehousing 

• Extraction: Access control. Agent technologies. Intelligent push 
technologies. Intent inferencing. Dynamic access control. Information 
usage analysis. User modeling 

• Decision Support: Structuring, Advice, Uncertainty portrayal. Tradeoff 
management 

• Aggregation and Fusion: Video compression. Wrapping legacy 
systems, Information synchronization. Geospatial and temporal 
indexing, Object extraction for compression. Meta-data language. Data 
fusion. Information life cycle. Database mediation. Rapid knowledge 
formation 
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• Accessing: Collaboration technologies. Self-healing networks. Multi¬ 
level access (e.g., security). Parallel access for speed 

• Labeling: Uncertainty, Domain-specific taxonomies, ontologies 

• Understanding: Dynamic situation modeling, Sensitivity analysis 

INFORMATION ACCESS Technologies: 

• Routing: Internet, ATM switching, broadcast technologies, etc.. 
Dynamic bandwidth management 

• Transmission: Assured delivery. Non-repudiation 

• Perception: 3-D Visualization, Natural language, Non-traditional 
senses. Drilldown, 3-D audio. Tailoring 

• Protection: Encryption, Recall of inaccurate information 

• User modeling: Info needs models. Dialog management. Context 
understanding, Intent inferencing 

• Communication: Conversational query. Speech, Natural language. 
Annotation, Domain-specific gesturing 

• Collaboration: Sharing, Advanced white-boarding. Domain-specific 
workflow management. Mixed initiatives. Facilitation 

As an example of how a listing of technologies can contribute to an 
understanding of the criteria needed for this analysis, consider a Decision 
Support System (DSS). DSS are computer-aided tools that emulate the 
reasoning process of a human expert making decisions. DSS are currently 
employed in the fields of business, medicine, law, and environmental sciences. 
For this project, tools are needed that support decision making. A DSS 
framework is designed to analyze “what-if scenarios and help make decisions 
that involve multi-criteria inputs. 

DSS are classified into categories based on the services and features 
supported. For purposes of this project, SRA considered knowledge-driven DSS 
and model-driven DSS. Knowledge-driven DSS are pre-programmed with 
specialized-problem expertise. They contain knowledge about a specific domain, 
and use this knowledge to aid the decision process of the user. The benefit of 
this type of DSS is that domain experts can “train” the tool in the initial stages 
and prepare it so that a user need not be a domain expert. The drawbacks are 
that training of the tool is cumbersome, and it may not be very adaptive to 
changing scenarios. 

Model-driven DSS (S/AM™ software is an example) use statistical models to 
support decision-making. These DSS technologies rely on the data provided by 
the user (decision tree, weighting of the nodes) for analysis. On the positive side. 
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no overhead of training the tools is required, and these DSS tend to be adaptive 
to changing scenarios. The main drawback is that any given user must be a 
domain expert to apply the tool effectively. 

While this is just one example of how a knowledge of the technologies can 
be applied to develop criteria for tool analysis, one trend should be made clear: 

AH of the technologies evaluated for this project will have 
advantages and disadvantages, so this meant it was critically 
important to ensure customer needs were well understood and 
that the customer was involved in all aspects of evaluation and 
selection criteria development, prioritization, and weighting. 

As a further illustration of how we used an understanding of relevant 
technologies to select and prioritize criteria, refer to Table 1. This table presents, 
for two key categories of tools (Link-Node Analysis and Decision Support 
Systems), potential techniques used, concepts and theory applied, concerns, and 
a candidate set of criteria that could be used to evaluate the tool. 


Table 1. Illustration of Understanding Technologies 


Categories 

Link - Node 

Analysis 

Decision Support Systems (DSS) 

Technique Used 

Link Analysis - Finds 

and represents 
patterns between 
different variables in 
the data sets 

Model - driven 
(Decision Analysis Tools) 

Knowledge - driven 

Concepts/Theory 

Applied 

Data Mining? 

Statistical approach 
(Bayesian Networks) 

Case Based 

Reasoning (CBR) 

Rule Based 

Reasoning (RBR) 

- “Learning” concepts 

Concerns 

How do the tools 

decide which 
variables to use to 
discover patterns? 
Does the user specify 
these variables? 

What other 
information does the 
system need? 

What is the output; 
Sensitivity Analysis or 
Impact Analysis 

For CBR 

Similarity Function 
Indexing/Retrieval 
technique 

For RBR 

Experts define rules 

Use data-mining to 
generate rules 

Criteria 

Interactivity with 

the user 

Representation 

aiding 

Ability to interface 
with multiple 
databases 

Interactivity with the 
user 

Collaborative support 
Qualitative vs. 
Quantitative weights? 
Assessment utilities 
used 

L Interactivity with the 

user 

Predictive capability? 
Evaluate data-mining 
tools? 
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There are some key terms that will keep coming up in this report and in the 
attachments. They include: 

• Probability Theory 

• Decision/Utility Theory 

• Bayesian/Belief/Causal/Probability Nets 

• Influence Diagrams 

• Decision Trees 

• Criteria model 

Definitions of these terms and a tutorial briefing are provided as Attachment 
4. This information can also be found in the Project Glossary (Attachment 11). 

After completion of our requirements analysis, breaking down the functions, 
sub-functions and technologies within the project framework, and in-depth 
discussions with our customers, the SRA team established the criteria to 
evaluate candidate modeling tools. Starting at a macro level, we sought 
capabilities that fit the following criteria: 

• Bayesian-based/inferential design, to permit probabilistic, or levels of 
confidence, outputs 

• Analyst-friendly and (relatively) quickly trainable operation 

• Links and nodes derivation and representation, especially for 
“weighting” of the (influence) links and (individual/group) nodes 

• Relational data/knowledge base utilization and applications 
functionality 

Detailed Criteria, developed from a thorough understanding of the mission 
requirements fell into four criteria categories: 

• General Features 

• Input Manipulation 

• Performance Extensibility 

• Interoperability 

General Features included criteria such as ease of use (which was, in turn, 
analyzed in further detail with more detailed, tailored criteria), cost, maturity, 
targeted industries, major clients, type of operating system, recommended 
hardware, and architecture. 
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• Input Manipulation criteria investigated type of network support, model 
building capability, type of graph support, conditional probability table 
specification, model validation, inference algorithm used, analysis techniques, 
and ability to limit computational complexity. 

Performance and Extensibility criteria included the language used for 
development, availability of source code and API, whether a benchmark was 
used to evaluate tool performance, maximum number of nodes supported, and 
error recovery support. 

Finally, the Interoperability category looked at criteria such as ability to 
interface with databases, formats used to save models, ability to export models 
and analysis results to other applications, use of open architecture standards, 
and capability for group collaboration. 

The complete set of evaluation criteria and results from surveys of selected 
vendor capabilities are captured in Criteria/Data Capture Sheets, a set of 
worksheets presented in Attachment 3. 

4.3 TASK 2: Market Survey 

Task : Compile a comprehensive list (within resource constraints) of capabilities 
that could satisfy some or all of the modeling requirements of this project. 

Task 2 Deliverables : Excel spreadsheet listing all candidate tools with notes on 
selection and elimination criteria; list of contacts and sources of information 
(Attachment 3); briefing on how we prioritized candidates for detailed data 
collection and assessment (Attachment 5). 

Using contacts identified by the customer, our knowledge of the market, and 
some additional investigations using keyword searches of relevant databases 
and the world-wide-web, the SRA team developed an extensive list of candidate 
tools. This list included over 80 candidates - about fifty of these were judged to 
be too much in the research phase (too immature) to be reliably evaluated. The 
remaining (30+) tools were evaluated through another screen - whether they 
were directly applicable to the customer’s problem domain. 

The SRA team decided to group the tools based on the underlying technique 
used for decision modeling. The tools were categorized into four major 
categories: Bayesian Networks, Influence Networks, Decision Trees and Criteria 
Modeling (i.e., building hierarchies of criteria and ranking them using AHP or 
SMART techniques). 

The following criteria helped prioritize the class of tools we would carry into 
the next task/phase (Task 4): 
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IMPORTANT : Tools that support Bayesian and Influence nets, permit 
manual construction of nets, and support evidence entry and belief 
updates (diagnostic inference). 

NOT IMPORTANT : Tools that are add-ins to other applications (ex: 
spread sheet add in), tools that rely on either decision tree technique or 
Multi-attribute Utility Theory (MAUT) (the problem domain is too big to be 
represented as one of the above), or tools that provide data mining 
capabilities given large amounts of data (NASIC analysts may not have 
large amounts of data in the early stages of implementation). 

Tools that used Decision Trees were not considered because we wanted 
tools that represented decision models compactly. Decision Trees require explicit 
representation of every possible alternative and this would soon result in a model 
that is not manageable in size. Criteria Modeling tools were not considered, as 
they do not allow users to represent uncertainties of how one variable influences 
another variable. Also, the requirement of a strict ordering among the criteria 
made Criteria Modeling not applicable to the problem domain. 

We considered tools that used Bayesian Networks or Influence Networks for 
decision modeling as they allow for a compact representation of the model and 
probabilistic representation of uncertainties of the amount of influence one 
variable has on the other. For a detailed description of Bayesian Networks and 
Influence Networks, refer to Reference [4] (at the end of Section 4). 

Figure 3 outlines the decision process explained above and lists the tools 
considered: 



JEMMA 

SIAM 

WAE 

SAMPLE 

Figure 3. Market Survey Screening Process 
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4.4 TASK 3: Data Collection 

Task : Based on criteria from Task 1, the SRA team developed a questionnaire 
(Attachment 6) that asked vendors of the “chosen” tools (from Task 2) to 
characterize their product in terms of our evaluation criteria. Questionnaire 
responses (Attachment 7) were mapped to the Criteria/Data Capture Sheet 
(Attachment 3). This information was then used as input to the information 
analysis task/phase of the project. 

Task 3 Deliverables : Raw data collected from candidate tool vendors 
(Attachment 7); Excel spreadsheet summarizing results from surveys 
(Attachment 3). 

4.5 TASK 4: Data Analysis 

Task: Rigorous, objective assessment of simulations, models, tools (obtain 
demo SW/models/tools and evaluate against criteria from Task 1). Assess how 
application of tools will support behavioral influence operations. 

Task 4 Deliverables : Output from InfoHarvest Criterium DecisionPlus analytical 
tool in Excel format; report on results of analysis 

4.5.1 Multi-Criteria Decision Analysis: 

After collecting information on the tools (using the questionnaire), the 
Analytical Hierarchy Process (AHP) technique was applied to evaluate and rank 
the Bayesian tools. AHP is a Multi Criteria Decision Analysis (MCDA) technique 
where a hierarchy of criteria is developed. At the top level of the hierarchy is the 
final goal (which in this case was the best Bayesian tool satisfying most of our 
criteria). The next level of hierarchy consists of criteria directly affecting the goal. 
The subsequent levels of the hierarchy consist of subcriteria affecting the criteria 
above them. The last level of the hierarchy consists of the alternatives that are 
being evaluated against the criteria. This hierarchy represents the decision model 
used to evaluate the alternatives. 

Once the model is constructed, the next step is to assign weights to all the 
elements (criteria and alternatives) in the model. The weight of an element 
represents the amount of contribution/influence that element has towards the 
criteria above it when compared to the other elements in its level. 

For our analysis, InfoHarvest Criterium® DecisionPlus® 3.0 (CDP) was used 
to determine the best Bayesian tool among all the alternatives. CDP is a decision 
analysis tool that supports AHP and other MCDA techniques to help aid in 
decision making. CDP provides a GUI to build the hierarchies; the user develops 
the decision hierarchy and then, using the AHP or SMART analysis technique, 
ranks the elements in the hierarchy. A scale of five ranking gradations is 
supported by the tool to assign ranks to the elements. Ranks can be assigned 
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numerically or verbally. The following table (Table 2) summarizes the ranking 
scale available for direct AHP ranking in CDP. 


Table 2. AHP Ranking Scale in CDP 


Numerical 

Verbal 

100 

Critical 

75 

Very 

Important 

50 

Important 

25 

Unimportant 

0 

Trivial 


4.5.2 Criteria Selection & Ranking: 

The criteria selected were based on the questionnaire. Questions that every 
tool answered similarly (e.g., Does the tool provide GUI support for model 
building?) or those that had responses that were not quantifiable (e.g., Who are 
the tool’s major clients?) were not considered as part of our criteria. Subsets of 
questions from the questionnaire were selected as criteria for every category in 
the questionnaire. 

The decision model for choosing an optimal Bayesian tool was developed as 
a two stage model. The first stage involved developing decision models for every 
category in the questionnaire and ranking the alternatives within each category. 
These decision models were called the sub-goal models. The final model was 
developed by grouping the four sub-goal models together as criteria for the final 
goal. 

Once the sub-goal and final goal models were built, input from the customer 
was taken to assign preference ordering of the criteria and subcriteria within all 
the decision models. These preference orderings were then translated into the 
ranking scale provided by CDP 3.0. 

The following tables (Tables 3-7) summarize, for every category, the 
questions that were considered as criteria/subcriteria and their associated 
weights. Every table is followed by a CDP hierarchy diagram of the table (Figures 
4-8), showing the normalized relative weights of criteria and appropriate levels of 
subcriteria. 
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Category - General Features 


Table 3. General Features Criteria and Relative Weights 


Criteria 

Rel 

Wt 

Subcriteria 

Rel 

Wt 

Questionnaire questions 
used from General 

Features category 

Maturity 

2 

>5 years 

1 

Question 2. Introduced 
a subcriteria of less 
than or greater than 5 
years 



<5 years 

2 


OS Support 

3 

MultipleOS - 
Win, Linux, 
(and/or) 

Mac 

1 

Question 5. 

Architecture 

1 

Client- 

Server 

1 

Question 7a 
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Category - Input Manipulation 

Table 4. Input Manipulation Criteria and Relative Weights 


Criteria 

Rel Wt 

Subcriteria 

RelWt 

Questionnaire questions 
used from Input 

Manipulation category 

Decision Node Support 

3 



Question 1b 

Analysis Technique 

1 



Question 7 

Mixed Graph Support 

3 



Question 3c 

Probability 

1 

Equations 

2 

Question 4 (b,c,d) 



Learning 

3 




Other 

3 




Noisy-Or 

1 

Information for this was 
gathered later on 

Virtual Evidence Support 

2 



Information for this was 
gathered later on 

Inference Algorithm 

2 

Exact 

1 

Question 6 



Approximate 

2 


Structure Learning 

3 



Question 2b 



Figure 5. Normalized relative weights for Input Manipulation criteria 
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Category - Performance and Extensibility 


Table 5. Performance and Extensibility Criteria and Relative Weights 


Criteria 

Rel Wt 

Subcriteria 

Rel Wt 

Questionnaire 
questions used 
from Performance & 
Extensibility 
category 

API Availability 

3 



Question 3 

GUI 

2 

C/C++ 

1 

Question 1a 



Java 

2 


Computational Engine 

1 

C/C++ 

1 

Question 1b 



Java 

2 




^.125 API AvailabilHv 1- 


liJNIO Perform&Extensibiltv 


^^.125 JavaGui | 


—I0J7S GUI h 

-10250 C++Gui 1 





CompEnqlne k 

c -10.167 Java I 





Figure 6. Normalized relative weights for Performance & Extensibility 

criteria 
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Category - Interoperability 


Table 6. Interoperability Criteria and Relative Weights 


Criteria 

Rel 

Wt 

Subcriteria 

Rel 

Wt 

Questionnaire questions 
used from Interoperability 
category 

Database Input Access 

2 

Use ODBC 

1 

Question 1(a,b,c) 



Direct 

3 




Save Models 

2 




SQL Interface 

3 


Export Analysis Report 

1 



Question 4 

OpenModelFormat 

2 



Question 2 

Export/Import Model 

3 



Question 3. Modified 
question by grouping the 
ability to Import nets built by 
other applications. 



Figure 7. Normalized relative weights for Interoperability criteria 


24 










































4.5.3 Final goal - Selecting the optimal Bayesian tool 

Table 7. Summary Criteria and Relative Weights 


Criteria 

Rel Wt 

General Features 

3 

Input Manipulation 

1 

Performance & 
Extensibility 

2 

Interoperability 

2 



Figure 8. Normalized relative weights for summary criteria 


4.5.4 Ranking the Bayesian Tools: 

The tools were ranked against all the criteria/subcriteria in the four decision 
models. Ranks were assigned based on the information gathered about the tool 
and the extent to which a tool could perform the criteria. For example, with 
criteria like Maturity of tool greater than five years (in the General Features sub¬ 
goal model), tools were simply assigned 100% if they were in the market for 
greater than five years and assigned 0% for the subcriteria Maturity, less than 
five years. But for criteria like ExportAnalysisReports (in the Interoperability sub¬ 
goal model), although all the tools provided some capability to export analysis 
reports, there was a wide range in the extent to which it could be done. For 
criteria like these, we assigned scores based on the extent of capabilities 
provided by the tool and not just based on if the tool said yes to the criteria in the 
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questionnaire. The following tables (Tables 8-11) summarize the ranks assigned 
to the tools. Each table is followed by additional notes that explain in detail some 
of the scores assigned. 

Category - General Features 


Table 8. Tool Rank for General Features Criteria 


Tool Name 

Maturity 

OS 

Support 

Architecture 


> 5 yrs 

< 5 yrs 

MultipleOS 

Client-Server 

Analytica 

100 

0 

75 

75 

BayesBuilder 

0 

100 

0 

75 

BayesiaLab 

0 

75 

100 

0 

Dxpress 

100 

0 

0 

100 

Ergo 

100 

0 

100 

0 

Hugin 

100 

0 

100 

75 

Netica 

100 

0 

100 

75 

SIAM 

100 

0 

75 

100 

Bnet2000 

50 

0 

100 

0 


Additional Notes: 

- For subcriteria < 5 yrs, 

- BayesiaLab is given a score of 75 because we wanted to differentiate 
between tools that are about 4-5 years old versus tools that have 1-2 
years of market maturity. 

- Bnet2000 is given a score of 50 because this tool was only used within 
Charles River Analytics (CRA) Inc, and does not have any commercial 
maturity outside CF^. 

- For subcriteria Client-Server support, Analytica, BayesBuilder, Hugin and 
Netica are given a score of 75 because there is no built in feature for Client- 
Server Architecture. This is accomplished using an Application Program 
Interfaces (API). 

- For subcriteria MultipleOS, we were distinguishing between tools that 
supported different kinds of OS (Windows, Linux, Mac) versus that supported 
only one kind of OS. A score of 100 was given if the tool supported Windows, 
Linux and/or Mac. 

- Although Analytica supports more than one OS (Windows and Mac), it 
was given a score of 75 because it did not support Linux (Customer 
emphasized that the tool has support for Linux). 

- Although SIAM is meant to support Windows and Linux (since it’s 
developed in Java), its not tested on Linux. Due to this, it was given a 
score of 75. 
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Category - Performance and Extensibility: 


Table 10. Tool Rank for Performance and Extensibility Criteria 



Computational 

Engine 

GUI 

API 

Availability 


C/C++ 

Java 

C++ 

Java 


Analytica 

100 

0 

100 

0 

100 

BayesBuilder 

100 

0 

0 

100 

100 

BayesiaLab 

0 

100 

0 

100 

100 

Dxpress 

100 

0 

100 

0 

100 

Ergo 

100 

0 

100 

0 

100 

Hugh 

100 

0 

0 

100 

100 

Netica 

100 

0 

100 

0 

100 

SIAM 

0 

100 

0 

100 

0 

Bnet2000 

0 

100 

0 

100 

75 


Additional Notes: 

- For the tool performance availability criteria, scores were assigned based on 
the information that was made available to us by the tool vendor. 

- For criteria API availability, Bnet2000 was given a score of 75 because 
negotiation is involved. 

Category- Interoperability 


Table 11. Tool Rank for Interoperability Criteria 


Tool Name 

Databases 

Export 

Analysis 

Reports 

Open Model 
Format 

Export/ 

Import 

Models 


Direct 

ODBC 

Save 

Models 

SQL 

Interface 




Analytica 

0 

100 

100 

100 

100 

100 

100 

BayesBuilder 

0 

0 

0 

0 

25 

0 

100 

BayesiaLab 

0 

100 

0 

100 

100 

100 

100 

Dxpress 

0 

0 

100 

0 

25 

0 

0 

Ergo 

0 

0 

0 

0 

75 

75 

0 

Hugh 

100 

100 

0 

0 

50 

75 

100 

Netica 

100 

100 

0 

100 

75 

75 

100 

SIAM 

0 

0 

0 

0 

100 

100 

100 

Bnet2000 

0 

0 

0 

0 

75 

100 

100 


Additional Notes: 

- For the critoria ExportAnalyeieRoports Gcoroe were assigned based on the 
following: 

- If the tool provided a graphical display of what-if and inference results; 
capability to copy and paste the network to other applications (MS 
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Word, PowerPoint, etc) by a simple click and drag, it was a given a 
score of 100. 

- If the tool primarily generated textual reports and allowed the user to 
select and paste the results and the nodes from other networks, to 
other application (click and drag), it was given a score of 75. 

- If the tool allows the user to save the network and the results as a file 
that could be opened later, it was given a score of 50. 

- If the tool allowed the user to only save the case files, it was given a 
score of 25. 

- For the criteria Open Model Format, Ergo is given a score of 75 because 
the open format feature is not as efficient as saving the model in the 
proprietary format: Although Hugin does not support any standard formats 
(XML, XBN, BIF etc), it is given a score of 75 because the format it 
supports, .net, is exportable to many other Bayesian Net tools; Netica is 
given a score of 75 because the Open Model Format feature is 
implemented, but not yet made available to the users. 


4.5.5 Discussion of the Results: 

Results were generated in two phases. The first phase involved generating 
scores for the tools within the four sub-goal models - General Features, Input 
Manipulation, Performance & Extensibility and Interoperability. These results 
were then combined together into a final decision model, as seen in Figure 8. 
The four criteria in Figure 8 are essentially soft-links to the four sub-goal models. 
The results from the sub-goal models are propagated to the final model, where 
they are combined to generate final decision scores. 

For every decision model, CDP displays the results in a decision score 
window. A decision score window displays the decision score of each of the 
alternatives in the model in a horizontal bar chart. Each line shows the name of 
the alternative, the value of the decision score, and a horizontal bar reflecting the 
value of the decision score [1]. 

For further analysis of the decision scores, CDP provides a feature that 
displays a breakdown view of contributions from each of the criteria toward the 
decision score of the alternative. Following are decision score windows and 
contribution by criteria windows (Figures 9-16) for all the sub-goal models and 
the final model. 


29 





Category - General Features (GF) 


-Oaowonc StmnFtikaM- 



Figure 9. GF Decision Score 
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Category- Input Manipulation (IM) 


-Decmirt inpulManip«Jaliorw 



Figure 11. IM Decision Score 



I StructLearning 
H MixedGraphSupport 
B VirtualEvidence 
[~1 Probabilities 
H AnalysisTech 
H DecisionNodeSupport 
B InferenceAlgorithm 


Figure 12. IM Contribution by Criteria 
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Category - Performance Extensibility fPE) 



Figure 13. PE Decision Score 



32 



























































Category - Interoperability (10) 


-Deciaionf Intefoperabiily 



Figure 15. lO Decision Score 



I Databases 
H ExpAnalysisReports 
I OpenModelFormat 
I I ExportlmportModeis 


Figure 16. 10 Contribution by Criteria 
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4.5.6 Scores from integrating the four sub-models into a final model 

-Dacwon: OptimafTocU 


Alternatives 

Value 1 

Netica 

0.138 

Hugin 

0.156 

Ergo 

0.070 

express 

0.085 

BayesiaLab 

0.151 

BayesBuilder 

0.069 

Analytica 

0132 

SIAM 

0110 

BNet2000 

00891 



aoo 


Figure 17. Integrated Decision Score 



0.16 

0.14 

0.12 

0.10 

0.08 

0.06 

0.04 

0.02 

0.00 


I Link - InputManipulation 
I Link - Interoperability 
I Link • Perform&Extensibil 
r~l Link • GeneralFeatures 


Figure 18. Integrated Contribution by Criteria 

From the final goal decision score window (Figure 17), Hugin ranks as the 
optimal Bayesian tool among all the alternatives (though others are close). From 
the final goal contribution by criteria window (Figure 18), it can be further seen 
that the score from Input Manipulation criteria had the maximum contribution 
towards the decision score of Hugin. The reason for this is that the criteria Input 
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Manipulation carried the maximum weight with respect to the final goal of 
choosing an optimal tool and within the Input Manipulation decision model, it can 
be seen that Hugin ranked the highest. This gave Hugin a lead compared to the 
other tools. 

BayesiaLab and Netica are the next best performance alternatives. Once 
again, the reader can look at the contributions by criteria window to gain further 
insight into the contributing factors towards the final decision scores of these 
tools. 

4.5.7 Discussion of GOTS Tools 

4.5.7.1 SAIC 

SIAM 

Compared to the other commercial Bayesian tools, SIAM ranked 5*^ among 
the nine alternatives. Following are the reasons for its relatively low decision 
score: 


• Input Manipulation: One of the major drawbacks of SIAM is that it does not 
support Bayesian Inferencing. Bayesian Inferencing is where the user, 
upon collection of more information about certain nodes in the net, enters 
the information (evidence) into the net. Based on the new information 
entered, the tool updates the probabilities of other nodes in the network. 
Some of the other features that SIAM does not address the learning of 
probabilities from a sample set and learning of the structure of the net 
from a database. These features are useful to have if the end user wants 
the tool to work with the existing data. 

• Performance & Extensibility: In this category, SIAM ranked low because of 
the emphasis placed on the language used to develop the computational 
engine and the Graphical User Interface (GUI). The preferred language 
was C/C++ and SIAM was developed in Java. The lack of an Application 
Program Interface (API) for SIAM also contributed to its low score. 

4.5.7.2 Charles River Analytics 

Organizational & Cultural Criteria for Adversary Modeling - Bnet2000 

OCCAM is a decision aiding system that helps build models by employing 
Bayesian Network Techniques, Rule Based Reasoning and Social Net Analysis. 

The following steps outline how these techniques work together in building a 
decision aiding system. (Note: This high level information is gathered from talking 
to the developers of OCCAM and not from using the tool or reading any technical 
description of the tool): 
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• An Entity is a person or organization that the user is interested in 
modeling. An Entity has attributes. 

• Application data is collected for the attributes of the entity. 

• Rules are constructed using this application data and also the data is 
further analyzed using Social Net Analysis tools. 

• The next step is to build a Bayesian net using Bnet2000. Bnet2000 is an 
in-house Bayesian tool. Currently, Bnet2000 does not support automatic 
construction of the Bayesian net from the data available. So the net is built 
by domain experts who understand the causal relationships between the 
nodes. Beliefs for the nodes are recorded based on the results from rule 
based reasoning, social net analysis of the application data and other 
proprietary techniques. 

As OCCAM employs more than Bayesian Nets, we felt that it needs to be 
addressed separately and not be included in the list of alternatives being rated. 
However, we did include Bnet2000 to the list of alternatives. From the decision 
scores it can be seen that it did not rank high. The reason for low scores is partly 
due to the fact that we did not have an evaluation/trial copy to work with and had 
to entirely depend on the information from the questionnaire. Also, it might be 
that Bnet2000 did not address some features due to the fact that it is being used 
together with other Al techniques, where the capabilities from these techniques 
compensated for the features that Bnet2000 lacks. 

Based on the information gathered for OCCAM, it can be said that it seems 
to be addressing the domain specific needs of the customer. It also employs all 
the techniques that the customer is planning to apply to build a decision support 
system that helps model the influences impacting human behavior. 

Samole/Grade 

Sample is an architecture that provides support to the user to build models 
that apply Artificial Intelligence (Al) techniques. Some of the techniques 
supported are Bayesian Nets, Fuzzy Logic, Genetic Algorithms, and Expert 
Systems etc. Grade is a GUI development environment that helps the user build 
a model by combining the techniques provided by Sample and to later simulate it. 
At SRA, we haven’t had a chance to work with this architecture/environment tool. 
From the information we collected, we believe that it might be useful to the 
customer if they plan to employ multiple Al techniques other than Bayesian Nets. 
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4.5.7.3 AFRL/Metrica, Inc. 


Evaluation of Cross-Cultural Models for Psychological Operations: Cross-Cultural 

PSYOP Decision Support System 

This system was developed as a part of a study performed by Metrica, Inc., 
for Air Force Research Laboratory. The study was done in three phases, where 
the last phase led to the development of the Cross-Cultural PSYOP Decision 
Support System (CCP-DSS). Refer to [2] (at end of Section 4) for the details of 
the study. 

The CCP-DSS is a web-based system that merges several databases 
related to PSYOP supporting objectives included in the system. These data 
include the associated influencing factors for the objectives, background 
information about the individual factors and response data from previous users, 
detailing their assessments of factor influence and probable target audience 
response to specific hypothetical PSYOP scenarios [2]. 

This DSS was not considered as one of our alternatives because of the 
following: 

• Employs SMART technique and not Bayesian methods. We are primarily 
focused on tools that employ Bayesian techniques. 

• It is a Knowledge-driven DSS. These are systems that have specialized- 
problem expertise. The system contains knowledge about a specific 
domain and uses this knowledge to aid decision process of the user. The 
alternatives we are considering are not Knowledge-driven DSS. 

Although the tool is not applicable as one of the alternatives, the feature of 
building a database that contains data supporting PSYOP objectives and helping 
the user based on this information is a very useful technique. This technique can 
be applied to the Bayesian tools to help the user determine the a-priori 
probabilities. 

4.5.8 Future Extensions 

For the evaluation process of the tools, we did not consider features of the 
tools that were in the process of being implemented. However, we feel that it is 
important to mention these extensions so that the customer can decide 
accordingly - to go with the current optimal tool or to wait for a tool (that is 
relatively cheap) and will have certain features implemented in their next 
versions. In general, with the commercial tools, tool vendors were very flexible 
regarding extension of their feature set to accommodate customer needs. Table 
12 summarizes the tools and future extensions in their next releases. This 
information was gathered from talking to the respective tool representatives. 
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Table 12. Planned Extensions for Some Vendors 


Tool Name 

Extensions 

Netica 

Availability of 
OpenModel Format 
and Structure 

Learning 

BayesiaLab 

Availability of 

Decision Nodes and 
Conflict Analysis 

Ergo 

Sensitivity Analysis 
can be performed via 
GUI 


4.5.9 “Ease of Use” Evaluation 
4.5.9.1 Industry Standard Techniques 

The usability evaluation was conducted on the four demo version 
applications that ranked highest in the functional evaluation: Analytics, Netica, 
Hugin, and BayesiaLab. Three primary approaches to usability assessment were 
considered in determining an evaluative methodology for this project: 1) heuristic 
evaluations with usability guidelines, 2) cognitive walkthroughs, and 3) usability 
testing. The selected methodology included elements of all three techniques. 
Testing was conducted by a human factors researcher; the results were 
collapsed and loaded into the decision support tool to obtain a usability ranking. 

A short discussion of the three methods, followed by a discussion of the selected 
evaluation methodology, the ranking criteria, and the evaluation results follow. 

Heuristic evaluations employ general usability guidelines (testable design 
principles) that focus attention on design areas that have historically proven 
sources of user difficulties. A good heuristic evaluation attempts to balance 
consistency (maintaining standardization for ease of use) and complexity 
(allowing variation to support expert users) and assess the application in the 
context of its expected use. Table 13 lists ten commonly accepted usability 
guidelines, derived by factor analysis of 249 usability problems across 11 
projects [6]. Cost/benefit analyses have shown that the most effective results are 
obtained using four evaluators for a comprehensive result [7]. Problems with this 
method involve assessing how to deal with trade-offs and how to apply 
guidelines judiciously within the current context. 
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Table 13. Ten Commonly Accepted Usability Guidelines 


Visibility of system status. System oroyides 
status feedback to user. 

Recognition rather than recall. System keeos all 
options and support for actions yisible. 

Match between system & the real world. System 
employs familiar terms and concepts. 

Helo recognizing, diagnosing & recoyering from 

errors. System assists rather than confuses. 

User control & freedom. System offers ouick exit 
path from errors and undo/redo function. 

Flexibility & efficiency of use. System offers 
accelerators for expert users while aiding noyices. 

Consistency & standards. System uses 
consistent terms and yisualizations. 

Aesthetic & minimalist desion. System 
communications are releyant and cogent. 

Error oreyention. System desion helos user 
ayoid errors. 

Helo & documentation. System helos are concise, 
clearly stated, task-focused, and searchable. 


Cognitive walkthroughs employ a set of representative tasks; the evaluator 
performs the tasks, step-by-step, and documents each system interaction. In a 
formal cognitive walkthrough, the evaluator uses pre-prepared forms to 
document the typical user’s specific goals, tasks, and knowledge at each step. 
How the system interface appears and how it changes in response to user 
actions is also documented. This form of evaluation is based on the psychology 
of inexperienced users and evaluates the system for ease of learning and 
support for exploratory learning. Identification of the psychological bases for user 
difficulties (e.g., confusion due to inadequate function labeling or error due to 
insufficiently explained input options) also indicates possible solutions. Testing is 
based on four main questions: 1) Will the users try to achieve the right effect? 

2) Will the users notice the correct action is available? 3) Will the user associate 
the correct action with the desired effect? and 4) If the user performs correctly, 
will the user realize progress is being made toward task accomplishment? The 
problems with cognitive walkthroughs center around three points. They require 
in-depth knowledge of the task domain for adequate test task selection. They are 
time-consuming: strict protocols and copious documentation require time. They 
are also susceptible to a low-level problem bias; concentrating at the keystroke 
level, it is difficult to see the high level problems that yield higher payoffs. 

Usability testing involves the development of empirical test plans that allow 
observers to assess system support to real users in realistic scenarios, 
performing real-world tasks. Observers document user behaviors and evaluate 
performance effectiveness; they also may employ a “thinking aloud” protocol to 
access user thought processes. Interviews and questionnaires are also 
techniques used to capture user impressions. Studies support three users as a 
cost effective number of test subjects [7]. Problems with this method involve the 
development of appropriate scenarios and tasks, difficulties with subject 
availability, and interpretation of results from different user expertise levels. 
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4.5.9.2 Integrated Approach 

This project did not support the full-scale employment of any of the foregoing 
techniques. Neither multiple domain experts nor multiple usability experts were 
available: time considerations were also a limiting factor, constraining both test 
duration and model task development. To meet project needs, a rapidly 
obtainable, high-level usability evaluation was created, drawing on elements in all 
three techniques. The test plan created for this project was based jointly on two 
industry standard heuristics. Xerox Corporation’s Usability Analysis and Design, 
Heuristic Evaluation: A System Checklist (an expansion of the ten factors 
identified earlier) [8] and DoD’s Defense Information Infrastructure Common 
Operating Environment (DIICOE) User Interface System Checklist [9]. The two 
checklists were examined for overlaps and edited accordingly. A preliminary 
examination of the applications indicated that many of the low-level detailed 
checks were either not problematic or subordinate in importance to larger issues; 
the combined checklist was filtered for non-issues and subordinate issues to 
leave the items likely to represent critical design concerns. The resulting list 
incorporated “bellwether” issues from all of the major categories (Table 14). The 
matrix in Attachment 9 offers more detail. 


Table 14. List of Significant Evaluative issues 


No. Issue 

4b Error Prevention/Recovery 

4c System Shutdown 

5 Flexibility/Efficiency/Legibility 

6 Other Functionality 

7 Visualizations 

8 Print Control & Reports 

9 Heip/Documentation 

9a Type of Online Help file 

9b Information Organization 

9c Accessibility of Language 

9d Tutorial 


No. Issue 

1 User/Computer Interaction 
1a Icons/Push Buttons 

1b Menus 

1c Windows/Diaiog Boxes 

Id Feedback 

2 Consistency/Standards 

2a Windows Metaphor 

2b Common Look & Feei 

3 System Status Visibility 

4 Errors, Error Prevention & Recovery 

4a Errors 


Although grouped somewhat differently to better correspond with potential 
problem areas in the applications, all but one of the ten factors were explicitly 
reviewed. “Match between system and real world” was not considered truly 
appropriate to the current evaluation. The terminology for Bayesian statistics and 
Bayesian net-building was domain-specific; in this case it seemed more realistic 
to check documentation for clarity of language and concept explanations. The 
lead issues (each with more detailed subcriteria) were incorporated in a matrix; 
each application was reviewed separately and results and comments were noted. 

The selected approach integrated elements from the other usability 
evaluation techniques as well. The concepts of test tasks and bi-level analysis 
(low level data entry as well as more high-level functional organization and task 
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support issues) were drawn from cognitive walkthroughs and the documentation 
structure of the effort, from observational documentation methods. In the 
absence of a formal test model, three methods were used. First, in keeping with 
the psychological orientation of the heuristic evaluation, the evaluator tried to 
exercise baseline network construction and compilation functions as a novice, 
looking for identifiable tools and intuitive processes. Difficulties were noted and 
assessed. Second, the evaluator attempted to build a small network and compile 
it. Third, the evaluator opened the Asia model (a diagnostic model included with 
each application as an example) and systematically altered it to observe the 
system response. All results were integrated in the relevant sections of the 
evaluation matrix. 

The initial documentation effort was not set up to be hierarchical. However, 
after reviewing the results, it was determined that MCDA AHP could be applied to 
the usability criteria as well. Accordingly, the documentation was examined for 
trends and for uniform scores. Where all applications were rated uniformly 
acceptable, the criterion was dropped from further consideration. Where all 
applications were rated uniformly unacceptable, the results were analyzed and 
the criterion was expanded appropriately to try to capture the cause of the 
problem. Trends were evaluated for contributory factors and the criteria list was 
amended to probe those issues. The amended list was decomposed so each 
criterion could be weighted appropriately. As all the factors in the final criteria list 
were considered critical to good user interface design, weights were distributed 
relatively equally across the criteria. The final evaluative criteria were 
reassessed for each program and any changes in results were integrated into 
both the initial and final criteria sets. 
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Table 15. Final Evaluative Criteria List 


1. User Support 
Dialog Box Help 
Find Function 
Wizards 
T utorial 

Example Scenarios 
Levei of Complexity 
Process Orientation 
Language 

Key Word Search Capability 
Help 

Language 
Glossary 
Searchable Index 
TOC Organization 
Completeness 


2. Error Management 
Validation Errors 

Warnings 

Explanations 

Automatic Repair Function 
Input Errors 
Warnings 
Explanations 
System & Fatal Errors 

3. Efficiency Factors 

Rapid Input/Manipulation Methods 
Repetitive Action Capabilities 
Align Tool 
Reverse Links Tool 
Windows Common Look/Feel 
Comprehensive Node Representation 
Model Description 


4. Visualizations 
On Screen 
Tables & Networks 
Graphs 
Text 

Error Logs 
Printing 

Page-break Preview 
Scale-to-Fit Capability 
Completeness 


The final criteria list (Table 15 above) excluded most Windows 
Metaphor/Common Look and Feel issues. There were few problems with the 
Windows metaphor or standardization of application elements: it was collapsed 
to a single entry under Efficiency Factors. Feedback was not an issue either 
because it was uniformly adequate or uniformly untestable with the available 
models. Icons were not a significant factor in user problems. Major trends were 
in application documentation and internal support to task accomplishment—both 
Judged critical application usability. Error messages offered some problems: 
system design was insufficient to eliminate errors, but without a more detailed 
model it was difficult to probe further. Print control and print capabilities were 
insufficiently supported; the total number of visualizations each application 
offered were difficult to document and display. Efficiency factors were less 
critical to task completion, but certain features (e.g., reverse links and rapid 
netbiiildino tnnk) wniilH hfi vpry important to laroe network construction. The 
weighted evaluation criteria and AHP results are discussed in the sections below. 
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4.5.9.3 Ease of Use Results (Tables 16-27) 

User Support Evaluation Criteria 


Table 16. Definitions for User Support Criteria 


User Support 
Dialog Box Help 
Find Function 
Wizards 
Tutorial 

Example Scenarios 
Level of Complexity 

Process Orientation 
Language 

Key Word Search Capability 
Help 

Language 

Glossary 
Searchable Index 

TOC Organization 

Completeness 


Description 

Offers interpretative help filling in data fields from within the dialog box. 
Permits modeler to search for a specific node in a large network. 
Facilitates performing routine or specialized action sequences. 

Tutorial provides example scenarios for all supported utilities. 
Scenarios have sufficient complexity to exercise all functions. 

Tutorial language displays a process rather than a procedural 
orientation (step-by-step instructions rather than overview). 

Language is simple, direct, and uses as little jargon as possible. 
Tutorial has internal keyword search capability to support finding 
specific instructions. 

Language is simple, direct, and uses as little jargon as possible. 

Help includes a glossary of terms to facilitate understanding of 
technical terminology. 

Help includes a searchable index of all key terms. 

Table of Contents is organized to support reasonable sequence of 
modeler activities (either working with or creating networks). 

All functions and required inputs are completely explained in Help. 
Tutorials offer supplementary method of obtaining information. 


Table 17. Rank Structure for User Support Criteria 


User Support 


Dialog Box Help 
Find Function 

Wizards 

Tutorial 

Example Scenarios 

Level of Complexity 

Proi;ebb Oii«iil<aLiuii 

Language 


Rank Definition 

0= No dialog box Help and can't open Main Help w/dialog box open. 
25=No dialog box Help but can open Main Help w/dialog box open. 
50=Dialog box Help links to Main Help table of contents. 

75=Dialog box Help links to appropriate topic in Main Help. 
100=Dialog box Help opens popup explanations/instructions. 

Y= 100 
N=0 
Y= 100 
N=0 


75 = Most 
100= All 

Note: Can't tell if all functions are included in this level of audit. 
Basic functions=50 
Complex functions=75 
Allfunctions=100 

Note: Can't tell if exercises all functions with this level of audit. 
Y= 100 

N-O 

Clear Language=100 
Somewhat Clear=50 
Not Clear=0 
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User Support 

Key Word Search Capability 

Help 

Language 

Glossary 

Searchable Index 
TOC Organization 

Completeness 


Rank Definition 
Whole document=100 
By Chapter=75 
By Page=25 

Clear Language = 100 

Somewhat Clear=50 

Not Clear=0 

Y= 100 

N=0 

Y=100 

Some search capability=50 

N=0 

Y= 100 

N=0 

Complete=100 
Reasonably Complete=75 
Somewhat Complete=50 
Not Complete=25 


User Support Results 


Table 18. Results for User Support Criteria 


User Support 

Analytica 

Netica 

Hugin 

BayesiaLab 

Dialog Box Help 

25 

50 

25 

0 

Find Function 

100 

100 

0 

100 

Wizards 

0 

0 

100 

100 

Tutorial 

Example Scenarios 

100 

50 

100 

50 

Level of Complexity 

100 

50 

100 

50 

Process Orientation 

100 

0 

100 

100 

Language 

100 

100 

50 

100 

Key Word Search Capability 

100 

75 

75 

100 

Help 

Language 

100 

100 

50 

0 

Glossary 

100 

0 

0 

0 

Searchable Index 

100 

100 

0 

50 

TOC Organization 

100 

100 

0 

0 

Completeness 

75 

50 

25 

50 


All applications were distinctly lacking in context-sensitive Help, a very 
important user support function. An ideal software application provides 
substantive assistance with all decision points; this includes directions for how to 
fill in all data fields. The criterion Recall vs. Recognition, one of the ten standard 
usability guidelines, stresses the importance of keeping support information 
visible; this is an important error avoidance technique. In the four applications 
evaluated the major need for context-sensitive Help was in dialog box 
interactions; therefore, the evaluation refers specifically to Dialog Box Help. 
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Dialog boxes require the user to fill in data fields to accomplish a task. While it 
would be impractical to keep all data field descriptions and input 
criteria/constraints visible, it is very possible to use hover popups or rightclick 
responses to provide the necessary support. The worst case scenario was 
shown in BayesiaLab, where not only is there no internal help, but the main Help 
function cannot even be accessed without closing the dialog box (a clear case of 
recall dependence). 

Three of the four applications included a Find function, permitting users to 
search for and locate specific nodes by name; no capability was identified in 
Hugin. This feature could become important in large, complex network 
constructions. In contrast, only Hugin and BayesiaLab offered any kind of 
Wizards— Hugin, for setting up structure and batch learning and BayesiaLab, for 
file importation. Wizards are interactive help utilities that guide users through 
potentially complex tasks, allowing novice users to perform at higher levels of 
expertise. Often implemented as a sequence of dialog boxes that prompt users 
to fill in required details, properly documented wizards are an excellent way to 
support multiple levels of user expertise. 

The Help and Tutorial tools are critical to usability. Help documentation 
ideally is designed around its potential use, either as an on-line support or as a 
print document; readability on-line is not the same as readability in print. On-line 
Help is very important because it answers user questions during task 
accomplishment. Printable manuals are very important because they allow the 
user to study the application (and by extension, task performance) for extended 
periods without access to a computer. Tutorials (which also can be on-line or 
printed) walk users through example tasks, increasing their expertise. 

Information in tutorials does not substitute for Help. When users want answers, 
they want them immediately, and they often cannot afford to search through 
linear tutorial presentations to obtain critical information. Therefore, every user 
action should be supported by the main Help. Tutorials should provide step-by- 
step instructions for both simple, novice-level tasks and complex, expert-level 
activities. Both the main Help and the tutorial should be organized around task 
performance, beginning with frequently performed actions, continuing 
progressively through less frequent activities. The language in all documentation 
should be clear, concise, and understandable by all levels of user. Domain- 
specific terms should be explained in context and in a glossary. On-line Help 
and tutorials should support key word searches. 

BayesiaLab suffered from very poor translation in its on-line Help, rendering 
it almost incomprehensible; its tutorial was much more understandable, however. 
Hugin used HTML files for both its on-line Help and its tutorial, rendering 
searches difficult. It also used highly technical language in places, failed to 

provide Q glocoary, offorod no index for rapid navigation, and its table of contents 

was organized around alphabetized menu and toolbar items rather than tasks. 
Finding task support through Hugin was extremely difficult. However, Hugin’s 
tutorial was very good in its provision of introductory material and representative 
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task instructions. Netica’s HTML-based tutorial was also difficult to search and 
displayed a procedural rather than a step-by-step process orientation (since it is 
still under development it is unfair to rate its completeness). None of the main 
Helps appeared able to stand alone; task accomplishment required lengthy 
searches of both Help and tutorials; Hugin was worst as its on-line Help 
appeared to lack much critical information. Analytica provided by far the most 
detailed documentation (the manual is 548 pages), including a glossary, index, 
error messages and a function list; their Help and tutorials were PDFs (making 
them both searchable and printable) with multiple examples of varying complexity 
and a process orientation throughout. Analytica’s Help provides functional 
information while its tutorial steps the user through building representative 
models. The only weakness encountered was an occasional need to use the 
tutorial to supplement Help—but required information was readily available. 


Efficiency Factors Evaluation Criteria 

Table 19. Definitions for Efficiency Factor Criteria 


Efficiency Factors 
Rapid Input/Manipulation Methods 

Repetitive Action Capabilities 
Align Tool 

Reverse Links Tool 

Windows Common Look/Feel 

Comprehensive Node Representation 
Model Description 


Description 

Facilitates rapid net-building by allowing user to 
create multiple nodes and connectors. 

Allows modeler to align multiple nodes. 

Allows modeler to reverse direction of the links 
between selected nodes. 

Uses a Windows metaphor and provides standard 
functions in standardized positions, stable tool 
bars, easily interpreted icons, hover descriptions of 
icons. 

Node network and node attributes and conditional 
probabilities can be displayed and edited on the 
same screen. 

Permits modeler to describe rationales and 
assumptions to aid others using the model. 


Table 20. Rank Structure for Efficiency Factors Criteria 


Efficiency Factors Rank Definition 

Rapid Input/Manipulation Methods 

Tool available=100 

Repetitive Action Capabilities Method available but takes 2 hands=75 

Y=100 

Align Tool N=0 

Y=100 

Reverse Links Tool N=0 

Very Consistent=100 (3 Ns or fewer in categories 
on master data collection sheet) 

Somewhat Consistent=75 (5 Ns in categories on 
master uata cullecUuii sheet) 

Seldom Consistent=25 
Not Consistent=0 

Windows Common Look/Feel 
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Efficiency Factors 

Comprehensive Node Representation 
Model Description 


Rank Definition 

Very Consistent=100 (3 Ns or fewer in categories 
on master data collection sheet) 

Somewhat Consistent=75 (5 Ns in categories on 
master data collection sheet) 

Seldom Consistent=25 
Not Consistent=0 
Y=100 
N=0 


Efficiency Factor Results 


Table 21. Results for Efficiency Factors Criteria 


Efficiency Factors 

Rapid Input/Manipulation 
Methods 

Analytica 

Netica 

Hugin 

BayesiaLab 

Repetitive Action Capabilities 

100 

100 

75 

100 

Align Tool 

100 

100 

100 

0 

Reverse Links Tool 

0 

100 

100 

0 

Windows Common Look/Feel 
Comprehensive Node 

75 

75 

100 

100 

Representation 

25 

50 

100 

100 

Model Description 

0 

100 

0 

0 


[jNude Edition 


All of the applications had repetitive action capabilities and all but 
BayesiaLab had an align tool; both capabilities speed model building—important 
when building very large networks. Netica and Hugin supported reversing links, 
another useful capability to speed the network-building process. A familiar 
Windows-like environment is also a factor that speeds task completion; all tools 
were acceptable, but Analytica used data 
field screens with cells and dropdown 
boxes for some of its data entry and these 
were harder to interpret and required extra 
cursor movements. The data cells were 
particularly confusing, as the default view 
showed a single cell and the user had to 
select that cell and hit the Enter key to add 
cells. This was explained in the tutorial, 
but was not supported in the data screen 
itself. Figure 19 shows BayesiaLab's 
composite dialog box that allows the user 
to rapidly enter all supporting node data. 
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Figure 19. BayesiaLab composite dialog 
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Error Management Evaluation Criteria 


Table 22. Definitions for Error Management Criteria 

Error Management Description 

Conditional probability errors (failure to sum to 1) and cyclic 
Validation Errors errors (node connections form cycle). 

Program warns user an error has occurred via visible or 
Warnings audible signals. 

Warning includes an explanation of the location and type of 
Explanations error. 

Program can be set to repair probability errors automatically 
Automatic Repair Function (automatically normalizes data). 

Input Errors Errors in types of information input into data fields. 

Program warns user an error has occurred via visible or 
Warnings audible signals. 

Warning includes an explanation of the location and type of 
Explanations error. 

System & Fatal Errors Warns of system & insufTicient memory errors. 


Table 23. Rank Structure for Error Management Criteria 


Error Management 
Validation Errors 

Warnings 

Explanations 

Automatic Repair Function 
Input Errors 

Warnings 

Explanations 

System & Fatal Errors 


Rank Definition 

Message Box=100 

Symbol w/popup=50 

Explains probability & cycles=100 

Explains cycles only=50 

Must set to normalize=100 

Warns will automatically normalize=50 

Automatically normalizes=0 

Boxed text warning=100 
Symbol=75 

Message pops up=100 

Takes 2 hands to display message=50 

Y=100 

N=0 


Error Management Results 


Table 24. Results for Error Management Criteria 


Error Management 

Analytica 

Netica 

Hugin 

BayesiaLab 

Validation Errors 

Warnings 

100 

100 

100 

100 

Explanations 

100 

100 

50 

50 

Automatic Repair Function 

50 

100 

0 

0 

Input Frrors 

Warnings 

100 

100 

100 

75 

Explanations 

100 

100 

50 

50 

System & Fatal Errors 

100 

0 

0 

0 
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This set of criteria, while very important to the user, was not thoroughly 
investigated. Factors involved in this decision included the variation in functions 
supported by the applications (e.g.. Analytics supported dynamic but not static 
cyclic dependencies, whereas Netica, Hugin and BayesiaLab flag and prohibit 
directed cyclic dependencies) and the lack of a representative, rigorous test 
model. The standard example, “Asia", included with each program, didn’t fully 
exercise the error management capabilities of the systems. 

All applications prohibited cyclic errors with an accompanying explanation 
{Analytica’s warning offered an alternative means of defining cycle nodes to 
make the cycle dynamic). Probability table error management was handled in 
three different ways: Analytics warned that it would normalize the illegal entry if 
directed to proceed with compilation, Netica simply flagged probability table 
errors, whereas Hugin and BayesiaLab automatically normalized values. 
Automatic normalization was judged a fault in this evaluation because it denied 
user control and hid the potential introduction of erroneous values from the user. 

Input errors were the most difficult to evaluate because the evaluator, with 

only limited knowledge of Bayesian net 
building and a limited test model, did not 
test all possible error conditions. 
Evaluation in this section was based on 
whether the system provided input error 
notification, how readily visible the 
warning was, the clarity of explanation 
and provision of instructions to repair the 
error. All systems provided some form of 
error message {BayesiaLab used a 
warning symbol that required effort to 
open): none provided specific 
instructions along with their explanations. 
This was considered a serious lack, 
given the complexity of the network¬ 
building task. See Figure 20 for sample 
messages. 



Hugin Error Message notes an evidence input error for a 
continuous chance node 



Anaiytica Error Message notes a probability input error, 
offering an autoroated fix or a chance to edit the input. 


Figure 20. Sample error messages 
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Visualizations Evaluation Criteria 


Table 25. Definitions for Visualizations Criteria 


Visualizations 
On Screen 

Tables & Networks 

Graphs 

Text 

Error Logs 
Printing 

Page-break Preview 
Scale-to-Fit Capability 
Completeness 


Description 

Displays node network and associated conditional probability tables 
simultaneously. 

Displays node information in different graph formats (bars, line graphs, 
etc. 

Displays text descriptions of node network. 

Displays user’s session error log in text box. 

Permits the modeler to see where the page breaks will occur in the 
network or table and adjust page setup accordingly. 

Permits the modeler to scale the network to fit the page. 

Prints nets, tables, graphs, text reports and error logs (or some subset). 


Table 26. Rank Structure for Visualizations Criteria 


Visualizations 
On Screen 
Tables & Networks 
Graphs 
Text 

Error Logs 
Printing 

Page-break Preview 
Scale-to-Fit Capability 


Completeness 


Rank Definition 


Each of the On Screen attributes rates 25 if Y and 0 if N. All 
four attributes sum to an On Screen score of 100. 


Y=100 

N=0 

Y=100 

N=0 

Prints everything it shows=100 (includes reports) 
Prints tables, network & text=75 
Prints tables & network=50 


Visualizations Results 


Table 27. Results for Visualizations Criteria 


Visualizations 

Analytica 

Netica 

Hugin 

BayesiaLab 

On Screen 

Tables & Networks 

25 

25 

25 

25 

Graphs 

25 

25 

25 

25 

Text 

0 

25 

0 

25 

Error Logs 

0 

25 

0 

0 

Printing 

Page-break Preview 

100 

100 

0 

0 

Goolo to Pit Capability 

100 

0 

100 

inn 

Completeness 

100 

100 

50 

75 


50 









Figure 21. Summary visualizations from each application 


Visualizations were divided into visual representations on screen and print 
capabilities. The applications varied widely in the representations offered and 
the forms taken. In Figure 21 above, Analytica’s summary visualization was 
programmed, while the others were available as menu options. The quality of 
the visualization varied considerably. Analytica did not support simultaneous 
representations of their network node maps, attribute charts, and conditional 
probability data. In contrast, BayesiaLab used an easily read and understood 
composite attribute/conditional probability table, visible with the node map, and 
Netica even displayed the network description and error log in frames. The 
ability to add a network description to aid other users was seen only in Netica 
and BayesiaLab] error logs (important to users learning the system) were 
available only in Netica. Print capabilities ranged from Hugin's restriction to 
networks, node attribute, and probability tables only to Analytica and 
BayesiaLab’s ability to print all visualizations and report results. Of the four 
applications, only Hugh had no sensitivity analysis capability. 

4.5.9.4 AHP Results 

As noted previously, the applications varied greatly in their intended use as 
well as their capabilities. The results of the AHP analysis (Figures 22 and 23) 
showed Netica the most user friendly in terms of user support, efficiency of entry, 
error management, and supported visualizations: Analytica was a close second 
with the other two applications ranking significantly lower in ease-of-use. 
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Figure 22. Usability Factors Decision Score 


Contributions to Usability from Level:Level 


0.35 


0.35 



B Efficiency Factors 
■ Visuaiizations 
I User Support 
□ Error Management 


Anaiytica 


BayesiaLab 


Hugin 


Figure 23. Usability Factors by Criteria 
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5 RECOMMENDATIONS 

5.1 Text Mining Tools 

As mentioned previously, we are considering three categories of tools - Text 
Mining, Investigative Analysis, and Bayesian tools that would help an analyst with 
his/her task. However, as the emphasis of this study was on Bayesian modeling 
tools, we have not had a chance to do an in-depth evaluation of text mining and 
investigative analysis tools. SRA, a regular user of vendor products, has 
gathered information from sources that do in-depth study of these tools. 

For text mining tools, we relied on market research done by SRA. Table 28 
lists well known tools and vendors in different categories of text mining. 


Table 28. Text Mining Tools 


Technique 

Tool Name 

Website 

Categorization of 
documents 

Autonomy 

http://wvvw.autonomy.com/Content/Technology/ 


Convera 

http://www.convera.com/Products/index.asp 


Entrieva 

http://www.entrieva.com/entrieva/index.htm 


InXight 

http://www.inxight.com/ 


Mohomine 

http://www.mohomine.com/ 


Stratify 

http://www.stratify.com/ 


Verity 

http://www.verity.com/products/index.html 




Summarization 

Copemic 

http://www.copemic.com/en/products/summarizer/index.html 


InXight 

http ://www. inxight.com/ 


5.2 Social Network Analysis Tools 

For social network analysis tools, the recommendation is based on the 
results from a detailed survey done for MITRE Corporation. In this report, i2’s 
Analyst Notebook Version 6 was the first recommendation and the next 
recommendation was Visual Analytics VisuaLinks. For further details regarding 
the criteria used, other tools evaluated and generation of results, refer to [3] (end 
of Section 4), included as Attachment 8 to this report. 
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5.3 Modeling Tools 
5.3.1 Performance 

From Section 4.5.6, it can be seen that Hugh ranks as the optimal Bayesian 
tool among all the ranked alternatives. The next best alternatives are BayesiaLab 
and Netica. 

Most of Hugh’s high decision score can be attributed to the number of 
features it supported in the category of Input Manipulation and the fact that 
criteria Input Manipulation was given the maximum weight towards the final goal 
of choosing an optimal tool. 

BayesiaLab has excellent design sense from a usability standpoint. Though 
there are some “ease of use” problems, they should be relatively easy to fix. 

Both Hugh and BayesiaLab personnel were very helpful and receptive to making 
changes and adding features in response to specific queries and customer 
requests (in fact the BayesiaLab folks have already made some 
changes/updates in response to queries from this study). 

Netica and BayesiaLab representatives are already addressing some of the 
performance issues that led to their ranking below Hugh (see Table 12). With 
the addition of these planned extensions, it is likely that the performance 
rankings for Netica and BayesiaLab will improve. 

SIAM is the only GOTS tool that we analyzed in detail, for reasons described 
in Section 4. One of the major drawbacks of SIAM is that it does not support 
Bayesian Inferencing. Other features that SIAM does not address; the learning 
of probabilities from a sample set and learning of the structure of the net from a 
database. These features are useful if the end user wants the tool to work with 
existing data. In Performance & Extensibility, SIAM ranked low because of the 
emphasis placed on the language used to develop the computational engine and 
the Graphical User Interface (GUI). The preferred language was C/C++ and 
SIAM was developed in Java. The lack of an Application Program Interface (API) 
for SIAM also contributed to its low score. All that said, SIAM does have 
performance features that will be useful for NASIC in the near term. 

Based on the information gathered for OCCAM, it can be said that it seems 
to be addressing the domain specific needs of the customer. It also employs all 
the techniques that the customer is planning to apply to build a decision support 
system that helps model the influences impacting human behavior. 

Sample might be useful to NASIC/BPB if they plan to employ multiple 
artificial intelligence techniques (in addition to Bayesian Nets). 

Although Metrica’s PSYOP DSS tool is not applicable as one of the ranked 
alternatives (does not use Bayesian methods), the feature of building a database 
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that contains data supporting PSYOP objectives (and helping the user based on 
this information) js a very useful technique. This technique can be applied to 
Bayesian tools to help the user determine the a-priori probabilities. 

5.3.2 Ease of Use 

Where Hugin and BayesiaLab were the top modeling tools from a 
performance perspective, Netica and Analytica stood out from those two in 
usability. Netica rated most “user friendly” in terms of user support, efficiency of 
entry, error management, and supported visualizations. Analytica was a close 
second in all these categories, with the Hugin and BayesiaLab ranking 
significantly lower in ease of use. 

SRA continues to receive information from all these companies. As we get 
more information, we will update our AHP analysis to ensure we have accurately 
modeled them and get the correct relative ranking. As we get results, we will 
provide them to the customer. However, from both a performance and a usability 
perspective, it would be far preferable to validate results from this study against a 
representative sample problem or scenario. 

5.3.3 Cost 

Table 29 summarizes the cost per single license, quantity licenses, and site 
license of the top five modeling tools. As some of the tools prices are listed in 
Euros, refer to httD://www.x-rates.com/calculator.html for an equivalent price in 
Dollars. 


Table 29. Cost Summary. 


Tool 

Pricing Structure 


(GUI and API Priced) 

Hugin 

1 License - 6300 Euros 

2 Licenses - 8390 Euros 

5 Licenses - 10,465 Euros 

10 Licenses - 12,550 Euros 

Site License - 16,785 Euros 


(GUI and API Priced) 

BayesiaLab 

1 License - 3450 Euros 

5 Licenses - 10,350 Euros 

10 Licenses - 17,250 Euros 

50 Licenses - 43,125 Euros 

Noiicsi 

GUI - $585 per License, API - $685 

Note 1; Site licenses available for 5 times as 
much. 

Note 2: API embedded is $20 to $175 
depending on the volume. 
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Tool 

Pricing Structure 

Analytica 

Professional - $1295 

Enterprise - $2495 

Analytical Decision Engine - $6000 

Note 1: Website (Attachment 3) gives 
comparison of features of these different 
versions) 

Note 2:15% discount for 10+ Licenses 

Note 3: 25% discount for 50+ Licenses 

SIAM 

All US gov't agencies hold a license to use 

SIAM for gov't purposes (just need federal ID) 


Complete cost information for the rest of the evaluated tools can be found in 
the “General Features” worksheet in Attachment 3. 

' 6 CONCLUSION 

6.1 NASIC’s Investment in Analytical Capabilities 

Many viable text/data mining options are available to NASIC/BPB. SRA is 
very familiar with NetOwl, and has used it for a variety of applications. However, 
other options were presented in Section 5. 

From discussions with experts in the field of Social Network Analysis, we 
were referred to a Mitre Corporation report (Attachment 9) that ranked i2’s 
Analyst Notebook Version 6 at the top of the list of these capabilities. The next 
best capability recommendation was Visual Analytics VisuaLinks. 

Based on the results of this study, NASIC/BPB has at least five viable 
options to meet mission modeling requirements: Hugin, BayesiaLab, Netica, 
Analytica, and SIAM. Unfortunately, the tool that ranked best in terms of 
performance (Hugin) is also rated, relative to the other tools, more difficult to use. 
BayesiaLab, another highly ranked tool in terms of performance, is also ranked 
lower in “ease of use." Netica and Analytica, while easier to use, did rank lower in 
performance. The lack of a clear “winner” is further complicated by the fact that 
some extensions to the performance capabilities of both Netica and BayesiaLab 
are planned in the next release of these tools. None of these options is out of the 
running in terms of cost/price, and (of the five top-rated performance options) 
SIAM has the advantage of being available for licensing free of charge to 
government customers. 

On balance, especially If ease of use is a critical factor for NASIC and the 
customer is willing to wait for the performance improvements (Table 12) planned 
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in the next release, Netica becomes a very attractive option. See Table 30 below 
for a summary of performance, usability and cost results. 


Table 30. Project Results Summary 



Performance 

Usability 

Cost 


GF 

IM 

PE 

10 

Overall 

Performance 

US 

EF 

EM 

VR 

Overall 

Usability 

Per 10 
Licenses 

Analytica 

e 

Y 

B 

B 

Q 

G 

G 

G 

Q 

G 

$51,000* 

BayesBuilder 

G 

R 

G 

R 

i 

NR 

NR 

NR 

NR 

NR 

4000 Eu 

($4,800)+ 
per seat 

BayesiaLab 

/ 

B 

e 

G 

B 

y 

i 

i 

< 

i 

17,250 Eu 

($20,700)+ 

Dxpress 

G 

/ 

B 

R 

/ 

NR 

NR 

NR 

NR 

NR 

Provided at 
negotiation 

Ergo 

/ 

R 

B 

/ 

Y 

NR 

NR 

NR 

NR 

NR 

? 

Hugin 

G 

B 

G 

G 

B 

{ 

0 

{ 

( 

< 

12.550 Eu 

($15,060)+ 

Netica 

G 

f 

B 

B 

G 

G 

B 

G 

G 

B 

$6,350** 

SIAM 

B 

f 

/ 

G 

G 

NR 

NR 

NR 

NR 

NR 

Free 

Bnet2000 

R 

i 

G 

G 

( 

NR 

NR 

NR 

NR 

NR 

Not 

provided 


Notes: 

B = Best relative to others G = Acceptable / = Marginal R = Worst relative to others 
* ADE version, assumes (single seat price X10) minus 15% discount for 10+ licenses 
** Price for site license (no price given for 10 seats) 

+ Conversion to dollars assumes 1.2 dollars per Euro (approximate current exchange rate) 

If NASIC/BPB wants further verification/validation of the results of this study 
before making an investment in a particular modeling capability, SRA 
recommends a more comprehensive evaluation using a sample or “benchmark” 
problem. 

6.2 Study Limitations 

There are some limitations on how these results should be used. An 
important factor when interpreting the results is to realize that the rankings for the 
tools were not generated based on building a Bayesian net using all the tools and 
then comparing the features. The ranks were primarily based on the responses 
from the tool vendors, information from the trial versions of the tools, and 
information on the web. In this process we might have overlooked some criteria 
that might prove useful when building a Bayesian model. Also, the final decision 
scores of the tools are a result of the criteria chosen and the weights assigned to 
them. The decision scores can change if a user evaluates the tools with a 
different set of criteria and different weights. 
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Another point to consider is that the SRA team worked with demonstration 
versions of these tools as they exist today. As planned extensions are added, 
and as more information is made available, these updates should be captured 
and considered in the analysis before finalizing an investment decision. 

6.3 Recommended Follow-On Activity 

SRA team members recommend further work to develop and implement a 
“benchmark” evaluation problem or scenario to verify the results of this study. As 
discussed above, tool rankings can be sensitive to both criteria and weighting. A 
sample problem or scenario that is representative of how the tools will actually be 
used by NASIC analysts can provide an additional level of insight and confidence 
to validate (or revise) these results. 

6.4 Interest from Industry and Government 

Many contacts (see Attachment 10) are interested in sharing the results of 
this effort. 
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7 Attachments 


1. Requirements Trace, Influence Operations and PEBO 
Architectures 

2. Analyst Questions 

3. Project Criteria/Data Sheets 

4. Definition of Terms and Tutorial Briefing 

5. Prioritization Briefing 

6. Market Survey Questionnaire 

7. Questionnaire Response Raw Data 

8. The Mitre Corporation Report 

9. Significant Evaluation Issues and Ease of Use Analysis 

10. POCList 

11. Project Glossary 
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Attachment 1 


Requirements Trace, Influence Operations 
and PEBO Architecture 
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Attachment 2 


Analyst Questions 
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MEMORANDUM FOR RECORD 


SUBJECT: 12 Sep 03 Analytical Capability for Behavioral Influence Operations Team 
Questions for Customer (Influence Operations Planners, Analysts) 

TO: NAIC/BPB (Eric Braeden) 

1. Eric, here is our first crack at a list of questions. Hopefully, this will continue our 
Task 1 progress toward a full understanding of how planners and analysts will use the 
tools. This will ensure we have the right criteria when we start evaluating candidates 
and alternatives in Tasks 2, 3, and 4. 

2. Questions focused on the initial planning phase: 

2.1. Describe your planning cycle for a representative/relevant RFI or problem. 

2.1.1. Who is basic Customer set and what do they typically ask you to provide? 

2.1.2. What Official Documents, TTP, Regs and checklists do you refer to for 
guidance? 

2.1.3. Do you use any worksheets for PSYOP planning and research? 

2.1.4. What Information Sources do you use? Do you use and have access to the 
following Databases and programs? 

2.1.4.1.Special Operations Command, Research, Analysis and Threat 
Evaluation System (SOCRATES) 

2.1.4.2. Psychological Operations Automated Data System (POADS) 

2.1.4.3. Foreign Publication Procurement Program (FPPP) 

3. Questions focused on the analysis process: 

3.1. Describe your PSYOP Target Audience analysis processes 

3.1.1. Type and Process 

3.1.1.1. Value analysis: Quantitative and Qualitative 

3.1.1.2. Nodal analysis 

3.1.2. What factors are taken into consideration? 

3.1.3. What are your target selection criteria? 

3.1.4. How do your targets get nominated and approved? 

3.1.5. How do you analyze secondary audiences for effects? 

3.1.6. What are a typical Timeframes for RFI’s, process and production? 

3.1.7. Do you track/monitor PSYOP operations? How? (i.e., PSYOP Campaign 
Control schedules with PSYOP intensity, objectives, temporal schedules) 

3.1.8. Do you currently conduct a form of “PSYOP weaponeering for “targets”? 

3.1.9. Is environmental analysis taken into consideration and used? (Climate, 
Weather, and Geographic) 

3.1.10. How are “targets” tracked in databases? (~ BE numbers) 

3.1.11. What type of role do you play in execution planning if any at all? 

3.1.12. Do you conduct PSYOP Effectiveness Assessment? 

3.1.12.1. Are MOEsi dcLcmiiiicU iiiid mciibuicd? Huw? 

3.1.12.2. Are Objectives not met by MOEs incorporated into future 
planning, and PSYOP assessment? 
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4. Questions that dig into what cognitive, cultural, organizational characteristics are 

valuable to the analyst: 

4.1. Points on decision making style, group goals and organizational behavior were 
noted; however, is it useful to include other culturally-based influences in 
cultural models used to test software capabilities? Should primary focus 
emphasize any one of the following? 

4.1.1. Cognitive characteristics? (e.g., idiocentric vs. allocentric orientation, 
mastery vs. fatalistic orientation, past vs. present vs. future time orientation, 
etc.) 

4.1.2. Institutional characteristics? (e.g., Muslim, largely agricultural/pastoral, 
tribal affiliations, oil-based international economy with wealth clustered in 
ruling family, etc.) 

4.1.3. Sensitivities? (e.g., gender roles, caste/class systems, social position of 
children/elderly, social taboos, etc.) 

4.1.4. Other? 

4.2. How do you incorporate cultural knowledge into your planning now? 

4.2.1. Is culture a critical thread in all influence planning, a means of fine-tuning 
non-cultural influences, the focal point of some planning efforts, etc? 

4.2.2. Are cultural issues always specifically included in influence operations? 

4.3. How do you foresee incorporating culture in the future? 

4.3.1. Do you expect to increase employment of cultural influences? 

4.3.2. Do you need cultural matrices that map associated strains of cultural 
influences and relate specific cultural influences to specific groups? 

4.3.3. Are there plans to collect specific data on effectiveness of cultural 
influence employment? 

4.3.4. Are there plans to try to separate/evaluate the relative effects of specific 
circumstances on effectiveness? 

4.4. Can you give several examples in which culture played a role in NAIC product 
development? 

4.4.1. What are some representative cases? 

4.4.2. What are some atypical cases? 

5. Questions focused on current tools that analysts use or are aware of: 

5.1. What type of Products do you currently produce? 

5.1.1. What are typical NAIC products like in format? (e.g.. Army PSYOP 
forms, multi-page text reports?) 

5.1.1.1 .How might cultural influences be incorporated in that format? (e.g., 
specifically identified, fully justified, or transparent to the user) 

5.1.2. What software do you use currently developing your products? 

5.1.2.1 .What works for you? 

5.1.2.2.What doesn’t work for you? 

5.1.3. What non-software supported techniques do you use currently to develop 
products? 

3.1.3.1 .Wlml Uiiptibililicb wuuld have helped piuduee diu:>e piudueL:^? 

5.2. What kinds of data base management systems are used now? 
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5.3. What is the nature of data that is useful to you now? (e.g., text, images, video, 
audio, etc.) Any expectations that this will change in the future? 

5.4. Are data visualization tools used? 

5.5. What types of decision modeling tools are used? 

5.6. What type of tools would help you better perform PSYOP analysis? 

6. We will be pursuing the answers to these questions through interviews with NAIC 
and other Influence Operations analysts over the next two weeks. Any assistance you can 
provide will be greatly appreciated. 


Michael L. Zywien, Principal 
Project Technical Lead 


cc: Greg Jarmarone 

Capt Tim Gameros 
Larry Daniel 
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Attachment 3 


Project Criteria/Data Sheets 
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Definition of Terms and Tutorial Briefing 
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Terms that keep coming 
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Terms in Detail 
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Conditional Probability Table (CPT) 

• CPT represents information about the effect of 
parent nodes on the child node. 
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Influence Diagrams 
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Example of Influence 
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Decision Trees 
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Criteria Model Example 
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Evaluating Networks 
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Attachment 5 


Prioritization Briefing 
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Discussion - Feedback 
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Task 5 (Demonstrations) eliminated 

Task 6 (Final report) - Draft outline available 








Methodology 
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Report results to customer 









Macro Criteria 



c 

o 

S 

0 

a 

o 

o 

n 

(0 

c 

‘(5 


o 

3 

O’ 

O 

> 

mmm» 

n 

0 

■o 

c 

0 

> 

■o 

c 

0 


0 

0 

C 

< 

cm’ 


^ 0 
O 0 
*^■0 
>0 
= c 
0^ 

o§- 

® 5 

0.0 

8 |> 

eg 

■Si 

0 > 

„ 

0 c 
0^0' 
O — 

k. 7 

ac 
0 0 

■o^ 

i:£ 

Is 

l§ 

® c 

■o.^ 

O 0 

2*^ 
C O 

■Os 
C O) 
0 C 

0‘’g 

C.O) 

“! I 

cos 


0 

c 

o 

■ ■M 

0 

O 

■q. 

o. 

0 

■o 

c 

0 

c 

o 

■ MB 

0 

N 


0 

0 

0 

.O 

0 

O) 

■o 

_0 

o 

c 

0 

** 

0 

■o 

0 

c = 
o 0 

_0 O 

0‘'S 




0 


3 0 ( 

O > 


<3 2. 


T5 i2 C- 

c o Op 

(B rs C 



o O Co 0 5 

osrJias 


101 








Elements of Decision Analysis 
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Questionnaire mapped to Excei Data Capture Sheet 
— Questionnaire mapped to spreadsheet 
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Market Survey 
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38 relevant research efforts 
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— Hugin Expert 

Many contacts are interested in sharing results of this effort 
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Attachment 6 


Market Survey Questionnaire 
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Tool Name: 

Website: 

Company Name: 

General Features 

1. Ease of use: (B) 

■ Is there feedback for processing delays? 

■ Do menu items and text descriptions use language that is commonly understood? 

■ Are menu items in standard places? Are layouts, terms, icon screen positions, windows, dialog boxes consistent? 

■ How many steps does it take to perform a frequently used action (desire not buried in menus; not multi-step)? 

■ Do you show user that he/she has made a mistake? How is user aided in correcting mistakes? 

■ Do you visually display system objects, actions, and options to users or are these available textually? How do you 
aid users’ memory for these choices? Are instructions easily available if the user forgets or loses his/her way? 

■ Does your system cater to novice and expert users? Allow users’ to create shortcuts for frequently used actions? 

■ How is help and/or documentation provided to the user? 

■ How does your software handle making best use of limited screen “real estate”? 

■ Top 5 “trouble” reports? 

2. Cost: Seat license; upgrades; support; maintenance; (for government customer)(C) 

3. Maturity of the tool: Niunber of years in the market, # versions/upgrades. (D) 

4. Targeted Industries: Business, medical, govt., etc (E) 

5. Major clients: (E) 

6. Type of Operating system supported: List all applicable OS (F) 

7. Recommended Hardware: (list both minimal and optimal) (G) 

a. Processing power: 

b. RAM: 

c. Other: 

8. Architecture support: (H) 

a. Client - Server Model: Yes I I No I I 

b. Desktop: Yes I I No I I 


no 








Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes Q No Q (B) 

b. Decision Networks/Influence Diagrams: Yes O No Q (C) 

c. Both Bayesian Nets and Influence Diagrams: Yes O No Q (D) 

d. Decision Trees: Yes [U No O (C) 

2. Model Building capability: 

a. Using GUI: Yes □ No □ (E) 

b. Other: Using a custom language, batch mode etc (F) 

c. From databases (i.e. Structure learning): Yes O No [H (G) 

3. Type of graph support: 

a. Only Undirected graph: Yes O No Q (H) 

b. Only Directed graph: Yes n No O (I) 

c. Chain graphs (i.e. mixed undirected and directed): Yes O No Q (J) 

4. Conditional Probability Table: 

a. Specify Tables: Yes Q No O (K) 

b. Specify Equations: Yes O No [U (L) 

c. From databases (learn from training sets): Yes Q No O (M) 

d. Other: Other Input techniques used (N) 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes Q No O (O) 

b. Cyclic Dependency structure check among variables: Yes O No O (P) 

6. Inference Algorithm used: List all relevant algorithms (Q) 

7. Analysis techniques provided/supported: List and explain (R) 






8. For tools using Bayesian Networks (Ref. Col B data) are there built in methods to 
trim the network or otherwise limit computational complexity with “relevance 
reasoning” or any other method: Yes Q No Q (S) 

a. If yes, explain: (S) 

Performance and Extensibility 

1. Language used to develop: 

a. GUI: (B) 

b. Computational Engine: (C) 

2. Is source code generally available to a government end user: Yes Q No O (D) 

(If no, is there any circumstance where the source code could be made available?) 

3. Is the API available to the end user: Yes Q No Q (E) 

4. Availability of the tool’s performance measurement: Yes Q No Q (F) 

a. If yes, please indicate the benchmark/procedure used to measure performance: 

5. Maximum Nodes a single model can support: Max nodes a model can support when 
rurming on the recommended system (G) 

6. Error recovery support (if something crashes, what happens?): (H) 


11 2 




Interoperability 


1. Ability to interface with databases: Yes Q (B,C) No Q 

a. If Yes, 

i. Using Open Data Base Connectivity (ODBC): Yes Q No Q (C) 

1. If Yes, list the data bases the ODBC driver supports: 

ii. Databases it can interface directly with (List them): (B) 

b. Does it support SQL interface: Yes Q No O (D) 

c. Can models created in the tool be saved to a database: Yes O No Q (E) 

2. Format used to the save the model: (E) 

3. Export the model to other applications: Yes I I No I 1(F) 

a. If yes, list the applications the model can be exported to: (F) 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes Q No D (G) 

a. If Yes, list applications that it can be exported to: Eg: PPT, Word, JPEG, PDF 
format, PS format, etc (G) 

5. Tools uses open standards: Yes Q No O (H) 

a. If yes, explain: (H) 

6. Provide group collaboration on model building and analysis: Yes O No Q (I) 

(e.g., can nodes be added as “sub-models” created by another user?) 
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Attachment 7 


Questionnaire Response Raw Data 
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Tool Name: Analytica 
Website: www.Iumina.com 

Company’s Name: Lumina Decision Systems, Inc 

General Features 

1. Cost: Professional $1295, Enterprise $2495, ADE $6000 (see Web site for 
lower prices until Nov 21 2003). 15% discount on 10+ licenses 25% discount 
on 50+ licenses. 

2. Maturity of the tool: Mature. First released 1996, now in release 3.0 

3. Targeted Industries: General: including technology, aerospace, consumer, 
defense, healthcare, energy & environment, higher education 

4. Major clients: Thousands. See www.lumina.com/ana/customers.htm 

5. Type of Operating system supported: Windows 95,98, NT 4.0,2000, XP, Mac 
OS up to 9.2 

6. Recommended Hardware: 

a. Processing power: Pentium or later, or Power PC 

b. RAM: 64MB 

c. Other: Color XGA screen 

7. Architecture support: 

a. Client - Server Model: Yes ^ No Q 

b. Desktop: Yes ^ No I I 

Additional Comments: Client-server model supported by Analytica Decision 
Engine. User must write end-user code. Analytica Web Publisher with automatically 
generated Web-based client user-interface expected release Q1 2004. 
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Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes 1^ No [~l 

b. Decision Networks/Influence Diagrams: Yes ^ No Q 

c. Both Bayesian Nets and Influence Diagrams: Yes No D 

2. Model Building capability: 

a. Using GUI: Yes 13 No □ 

b. From databases(i.e. Structure Learning): Yes Q No 3 

c. Other: 

3. Type of graph support: 

a. Only Undirected graph: Yes [3 No Q 

b. Only Directed graph: Yes 13 No f") 

c. Chain graphs (i.e. mixed undirected and directed): Yes O No Q 

4. Conditional Probability Table: 

a. Specify Tables: Yes 13 No FI 

b. Specify Equations: Yes 13 No F"! 

c. From databases (learn from training sets): Yes [3 No ^ 

d. Other: Can provide general functional form of relations, including 
procedures to generate probability distributions 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes 3 No Q 

b. Cyclic Dependency structure check among variables: Yes 3 No [3 

6. Inference Algorithm used: Monte Carlo and Latin Hypercube 

7. Analysis techniques provided: Many, including parametric sensitivity, 
correlation importance analysis, optimization. Note that Analytic can handle 
dynamic models (variables changing over time) without having to replicate 
nodes for time period. This makes it vastly easier to build, faster to evaluate, 
and simpler to understand dynamic models than competing methods. 
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8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes I I No 1^ 
a. If yes, explain: 

Additional Comments: Analytica is designed for and extremely versatile and 
efficient for forward or causal reasoning with influence diagrams. It is limited 
for backward or diagnostic reasoning. 

Performance and Extensibility 

1. Language used to develop: 

a. GULC-H- 

b. Computational Engine: C-H-, Analytica scripting 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user: Yes 1^ No I I 

4. Availability of the tool’s performance measurement: Yes Q No ^ 

a. If yes, please indicate the procedure in measuring the performance: 

5. Maximum Nodes a single model can support: Tens of thousands, each with 
multidimensional values, comparable to millions for conventional 
representations 

6. Error recovery support: Yes 1^ No I 1 

a. If yes, explain: Extensive diagnostic and error handling, including the 
ability to continue execution with partially undeflned or infinite 
values. Interpretive mode makes it easy to find problems. Execution 
profiling (in Enterprise version) supports model optimization. 
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Additional Comments: Analytica provides its own general scripting language, 
which is very powerful especially for multidimensional modeling. Some users 
who want to do substantial diagnostic inference use Analytica UI to create 
models and export to Winbugs to evaluate them. 

Interoperability 

1. Ability to interface with databases: Yes ^ No Q 

a. If Yes, 

i. Using ODBC: Yes lEI No □ 

ii. Databases it can interface directly with (list them): Any that 
supports ODBC 

b. Does it support SQL interface: Yes 1^ No I I 

c. Can the models be saved to a database: Yes ^ No Q 

2. Format used to the save the model: XML file format 

3. Export the model to other applications: Yes ^ No O 

a. If yes, list the applications the model can be exported to: Databases and 
editors that support XML. Model may also be edited in any word 
processor. 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes 1^ No I I 

a. If Yes, list applications that it can be exported to: Can copy and paste, or 
OLE hotlink tables and graphs to Microsoft Excel, Powerpoint and 
many other common Windows applications. Can use Excel seamlessly 
for graphing results. Can save table results into tab-delimited files for 
viewing and analysis in Excel or most statistical and database 
packages. 

5. Tools uses open standards: Yes ^ No Q 
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6. Provide group collaboration on model building and analysis: Yes ^ No I I 

Additional Comments: Multiple users can collaborate in building models, and can 
distinguish inputs and scenarios by multiple users or experts by indexing. 

Additional Notes : Each variable or node in Analytica can represent a single 
value (number or text) or a multidimensional table. This facility, along with its 
Intelligent Array capabilities provide great convenience, flexibility, and speed 
for scaling up to handle large "industrial-scale" problems. 

See www.lumina.com/casestudies for sample applications 
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Tool Name: BayesBuilder 

Website: http://www.mbfys.kun.nl/snn/nijmegen/index.php3?page=31 
Company’s Name: Smart Research BV 

General Features 

1. Cost: 2000 Euro per license for GUI only. 4000 Euro per license including 
API 

2. Maturity of the tool: 4 years 

3. Targeted Industries: BayesBuilder was constructed for a research project that 
aimed to build a large scale model for medieal diagnostics. 

4. Major clients: Our main commercial client is SKF. They used BayesBuilder+ 
API to build a web-based diagnostic system for bearing failure analysis. See 
http://evolution.skf.com/gb/article.asp?articleID=471 

5. Type of Operating system supported: Win32 Release for Windows 95, Windows 
98 and Windows NT (4.0) - or higher - on Intel hardware. 

6. Recommended Hardware: 

a. Processing power: Pentium or faster 

b. RAM:32 megabytes 

c. Other: 22 Mb on harddisk; Minimum desktop areafor GUI: 800x600 
pixels. Color palette: 256 colors, VGA. 

7. Architecture support; 

a. Client - Server Model: Yes ^ No O 

b. Desktop; Yes 1^ No O 

Additional Comments: for client server model, one needs to purchase the API ( a 
library of JAVA calls). API runs independently of GUI. 
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Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes ^ No O 

b. Decision Networks/Influence Diagrams: Yes Q No ^ 

c. Both Bayesian Nets and Influence Diagrams: Yes fl No 1^ 

2. Model Building capability: 

a. Using GUI: Yes lEI No □ 

b. From databases(i.e. Structure Learning): Yes fl No 1^ 

c. Other: via API (i.e. JAVA) 

3. Type of graph support: 

a. Only Undirected graph: Yes CH No 13 

b. Only Directed graph: Yes 13 No I I 

c. Chain graphs (i.e. mixed undirected and directed): Yes CH No [3 

4. Conditional Probability Table: 

a. Specify Tables: Yes 3 No Q 

b. Specify Equations: Yes I I No 13 

c. From databases (learn from training sets): Yes Q No 3 

d. Other: 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes 3 No 3 

b. Cyclic Dependency structure check among variables: Yes 3 No D 

6. Inference Algorithm used: junction tree 

7. Analysis techniques provided: (1) standard conditioning of variables, and (2) 
computer of correlations and cross entropy between pairs of variables. 
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8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes I I No 1^ 
a. If yes, explain: 

Additional Comments: 


Performance and Extensibility 

1. Language used to develop: 

a. GUI: JAVA 

b. Computational Engine: C-H- 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user: Yes ^ No D 

4. Availability of the tool’s performance measurement: Yes D No ^ 

a. If yes, please indicate the procedure in measuring the performance: 

5. Maximum Nodes a single model can support: no built in constraint- depends on 
complexity of the graph (clique size limited to 20) . Several hunderds of nodes 
run smoothly. I expect 3000 to 10000 nodes is feasible in sparse models, but I 
never tested this. 

6. Error recovery support: Yes I I No 1^ 

a. If yes, explain: Java error messages 

Additional Comments: 
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Interoperability 

1. Ability to interface with databases: Yes I I No 1^ 

a. If Yes, 

i. Using ODBC: Yes □ No lEI 

ii. Databases it can interface directly with (list them): 

b. Does it support SQL interface: Yes I I No 1^ 

c. Can the models be saved to a database: Yes I I No 1^ 

2. Format used to the save the model: text file (ASCII), possibly encrypted. Other 
save formats are not supported. 

3. Export the model to other applications: Yes I I No 1^ 

a. If yes, list the applications the model can be exported to: 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes ^ No Q 

a. If Yes, list applications that it can be exported to: gif, jpeg 

5. Tools uses open standards: Yes ^ No Q 

a. If yes, explain: inference is standard junction tree (with the complete- 
neighbour heuristic for constructing the junction tree) 

6. Provide group collaboration on model building and analysis: Yes Q No ^ 
Additional Comments: 

Additional Notes ; an unrestricted GUI-version can be downloaded free of 
charge to evaluate BayesBuilder. See webpage at the beginning of the 
questionnaire. This version contains a tutorial and a manual of the tool. 
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Tool Name: BayesiaLab 
Website: www.bayesia.com 
Company’s Name: Bayesia S.A. 


General Features 

1. Cost: For about 10-60 licenses or is their a group license, etc 
1 License: 3450 Euros 

5 Licenses: 10 350 Euros 
10 Licenses: 17 250 Euros 
50 Licenses: 43 125 Euros 

2. Maturity of the tool: BayesiaLab is on the market since 2002, but the first 
version of the tool has been initially realized in an academic research center 
in 1998 

3. Targeted Industries: Industrial companies. Consulting companies. Information 
Technology companies. Pharmaceuticals companies. Research centers and 
Universities 

4. Major clients: EOF, Saint Gobain, Cap Gemini Ernst&Young, Sumitomo 
Pharmaceuticals, Parks Canada, CNRS 

5. Type of Operating system supported: All the OS having a Java Virtual 
Machine 

6. Recommended Hardware: 

a. Processing power: >=1 Ghz 

b. RAM: >=256 

c. Other: 
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7. Architecture support: 

a. Client - Server Model: Yes I I No 1^ 

b. Desktop: Yes 1^ No I I 

Input Manipulation 

1. Type of Network support: 

a. Bayesian Networks: Yes Rl No F] 

b. Decision Networks/Influence Diagrams: Yes ^ No □ 

When going thru the tutorial, I did not see any discussion on whether 
BayesiaLab supports decision node type. I did realize that it supports the 
concept of Cost. Is this what you had in mind when you answered Yes for the 
above question? Can BayesiaLab support decision node types? 

Indeed, BayesiaLab supports two kinds of costs: a cost for knowing the value of a 
variable (useful for the adaptive questionnaires) and utility nodes that allows 
qualifying states defined by node value combinations. But, we have not implemented 
the so called decision nodes. We manage such nodes with probabilistic nodes (with 
action as modality) to evaluate policies. 

c. Both Bayesian Nets and Influence Diagrams: Yes ^ No F] 

2. Model Building capability: 

a. Using GUI: Yes 1^ No I I 

b. Other: Using a custom language, batch mode etc 

c. From databases (i.e. Structure learning): Yes ^ No F] 

3. Type of graph support: 

a. Only Undirected graph: Yes O No ^ 

b. Only Directed graph: Yes O No ^ 

c. Chain graphs (i.e. mixed undirected and directed); Yes ^ No F] 

4. Conditional Probability Table: 

a. Specify Tables: Yes ^ No FI 

b. Specify Equations; Yes ^ No FI 

c. From databases (learn from training sets): Yes ^ No FI 

d. Other: deterministic mode 
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i. Deterministic mode => if a node is logical and, or, etc deterministic 
mode allows for this logical representation. 

Does BayesiaLab support Noisy-or nodes? 

It is possible to quickly define a CPT thanks to the Noisy-Or-Function that is 
available in the equation editor but we don’t have specific Noisy-Or nodes. 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes ^ No Q 

b. Cyclic Dependency structure check among variables: Yes ^ No n 

6. Inference Algorithm used: Junction tree, Likelihood weighting 

Does BayesiaLab support both causal and diagnostic inference? Also, does 
the user have a choice to decide which type of inference algorithm (exact or 
stochastic) to use? 

Yes, you can do inference in any directions, from symptoms to causes, from 
causes to symptoms and any combination of these inferences. The inference menu 
allows you choosing between the exact and approximate inference. 

Does BayesiaLab support negative and likelihood evidence (like Hugin and 
Netica) 

Yes, for both exact and approximate inference. 

7. Analysis techniques provided: 

* Arc Analysis: allows highlighting the importance of the arc with respect to 
the complete structure. The thickness of the arc is proportional to the 
importance of the probabilistic relation it represents in the total probability 
law. 

* Target Node Analysis: allows the visualization of the quantity of information 
brought by each node for the knowledge of the target node. The brightness of 
the squares appearing inside the nodes is proportional to this quantity of 
information 

* Target Modality Analysis: allows visualizing, for each node, two kinds of 
information relative to the target: the type of the probabilistic relation binding 
this variable and the target variable, and the information gain brought by each 
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node for the knowledge of the modality of the target node 

* Target Analysis Report: Textual report describing the context of the analysis 
(list of the variables that are observed when the analysis is carried out), the 
probability distribution of the target variable knowing the context, a list of 
nodes, sorted by descending order according to their relative contribution to 
the knowing of the target variable (the nodes that do not bring any 
information do not appear in this list), for each value of the target, list of 
nodes, sorted by descending order according to their relative contribution to 
the knowing of the target value (if the node has only two modalities, this list is 
identical to the preceding one), for each influencing node, description the 
modal value with respect to the context and to the observed modality of the 
target node (this modal value comes with its probability, this section allows 
establishing the profile of this target value), for each influencing node, 
description of the modal value when the target node is unobserved (but 
knowing the context, that makes it possible to define the profile when the 
target variable is unobserved), and a measure indicating the variation between 
the modal values of the two profiles. This value is measured only when these 
values are identical. A positive value indicates then an information gain to the 
knowledge of the node value when the target value is observed. 

* Adaptive Questionnaires: This functionality allows the automatic and 
dynamic activation of the monitors of the variables that bring information on 
the probability distribution of the target variable at lower cost. The monitors 
are sorted according to their relevance (ratio between information gain in the 
current context and the cost implied for knowing the value of the variable). 

* Questionnaire based on the knowledge of a particular value of the target 
variable: This functionality, similar to the previous one, differs only by the 
measurement of the information gain. This gain is no longer measured 
according to the total probability distribution of the target variable, but is 
completely centered on the knowledge of a particular value 

* Causal analysis thanks to the equivalence classes. Arcs that can be reversed 
without changing the probability law are undirected. It is then possible to 
choose an orientation that is automatically propagated to the network 
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(orientation of the compelled arcs, update of the conditional probability 
tables). 

8. For tools using Bayesian Networks (Ref. Col B data) are there built in methods to 
trim the network or otherwise limit computational complexity with “relevance 
reasoning” or any other method; Yes I I No I I 

a. If yes, explain: I don't understand if you are speaking about building the 
networks through data mining or if you are speaking about inference in 
large Bayesian networks. In the first case, our learning methods are all 
based on the Minimum Description Length score (equivalent to having an 
automatic threshold in the constraint based approaches for determining 
statistical independence). Concerning inference, it is indeed impossible to 
construct the junction tree for large networks that are highly connected. In 
that case, on has to use the approximate inference. 

Thru this question, we were trying to find out if the tool has the capability to limit 
its inference based on the type of evidence provided. For example, given an evidence 
for a certain node, the tool know that it does not have to update the entire net, but 
only update a sub-section of the net that will be effected by this evidence. Does 
BayesiaLab have such a feature? 

Not for the moment. 

Performance and Extensibility 

1. Language used to develop: 

a. GUI: Java 

b. Computational Engine: java 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user; Yes ^ No Q 
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4. Availability of the tool’s performance measurement: Yes ^ No Q 

Thru this question, we wanted to know if you had any results on performance 
(time it took to do inference) of the tool for some “typical problems”. Do you 
have any results that show the performance of the tool for any problems that you 
have modeled and used? 

Indeed we have such results on benchmarks but only for learning (scientific 
publications on our learning algorithms), but not for inference. 

a. If yes, please indicate the procedure in measuring the performance: The 

first global performance measure is the MDL score available thanks 
to the console. Otherwise, BayesiaLab offers various tools for 
measuring the performance when specifying a target node : 

* Total precision of the model: number of correct predictions of the 
value of the target variable / number of cases in the data base 

* Confusion matrix: the total precision is useful but can be too 
general. The Confusion Matrix proposed by BayesiaLab allows 
having a more precise feedback about the model performances: 
reliability (i.e. the rate of correct prediction for each value given by 
the model), precision (the rate of detection for each value of the base) 

* Lift curve: represents the detection rate of the target value (Y-axis) 
with respect to the number of processed cases (X-axis) based on the 
order defined by the model. Allows choosing the best threshold to 
reach a particular rate of detection 

* ROC curve: True Positive Rate (Y-axis) against the False Positive 
Rate (X-axis). Allows choosing the best threshold to reach a particular 
ratio between true and false positives. 

5. Maximum Nodes a single model can support: As I said it before, it is difficult to 
give a maximum number of nodes since it depends also on the number of 
node values. However, last week, I have worked on a genetic data base 
containing 2000 columns with 3 modalities for each gene on a 1 GHz PC with 
256 MO without problem. 


129 







6. Error recovery support: Yes 1^ No I I 

a. If yes, explain: Exceptions throwned by the Java Virtual Machine are 
catched and it is possible to automatically send the error message to 
Bayesia so that we can correct the error. If the exception is relative to 
memory, a specific window explain what parameters have to be 
change to allow BayesiaLab to work with more memory. 


Interoperability 

1. Ability to interface with databases: Yes Q No ^ 

a. If Yes, 

i. Using ODBC: Yes 13 No □ 

1. If Yes, list the data bases the ODBC driver supports: 

ii. Databases it can interface directly with (list them): 

b. Does it support SQL interface: Yes 13 No I I 

c. Can the models be saved to a database: Yes Q No 3 

2. Format used to the save the model: XBL, a specific format based on XML. 
BayesiaLab can also read BIF and NET networks. 

3. Export the model to other applications: Yes I I No 13 

a. If yes, list the applications the model can be exported to: 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes 13 No I I 

a. If Yes, list applications that it can be exported to: It is possible to paste 
the monitors to any applications that interact with the clipboard of 
the OS. Monitors can be pasted as texts, arrays or images. 
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5. Tools uses open standards: Yes 1^ No I I 

a. If yes, explain: Bayesian Interchange Format (BIF) 


6. Provide group collaboration on model building and analysis: Yes ^ No O 
Can you please explain how group collaboration is supported? 

We have two tools that help working in collaboration on model building: 

1. It is possible to associate color tags to nodes in order to group them visually 
and to ease the brain storming with the domain experts. 

2. Its is possible to associate hypertext comments (with links to files or http 
address) to the network (by default the Author name and creation date) and to 
each node (to describe the role of the node, to increase the traceability, to 
justify the probabilities, to associate illustrations, schemas,...) 


Additional Notes : BayesiaLab also offers: 

* Complete wizards for importing data bases with complete missing value 
processing (filtering, replacing with given values, structural Expectation- 
Maximization), filtering capabilities (variables and values), intelligent 
discretization tools, database transposition (e.g. for the micro-array data 
processing). 

* Possibility to generate cases with respect to the current network 

* Possibility to save the database as associated to the current network, i.e. with 
its treatments (discretization, filtering, missing values processing) 

* Complete structural learning toolbox: three unsupervised learning methods to 
discover all the probabilistic relations that hold in the database, five supervised 
learning methods to characterize a target variable (from naive to markov 
blanket learning), and clustering methods that allow to segment the data. All 
these learning algorithms support missing values. 

* Hidden variables: it is possible to add hidden nodes, i.e. nodes that have not 
data in the database and to combine them manually/automatically with the 
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* Possibility to do structural learning from an existing network (a priori 
knowledge), and to fix the arcs so that the learning algorithm cannot change this 
knowledge. 

* Batch exploitation: Possibility to use the current network to do an off line 
tagging of the cases contained in a database. 

* Complete Dynamic Bayesian Networks processing to introduce the temporal 
dimension in the model (step by step simulation, period simulation, probability 
evolution in a graph that can be printed or saved, and possibility to associate an 
observation file). Possibility to use a time node, i.e. a parameter node that 
represents the time and that can be directly used in the equations describing 
probability distributions. 

* Constraint nodes: Boolean nodes that are always observed and that are used to 
express constraints between nodes 

* Automatic positioning of the nodes 

* Warning tags on the nodes that have a not a filled conditional probability table 

* Error tags on the nodes that have an incorrect equation due to parent 
modifications 

* Hypertext comments on the networks and the nodes 

* Color tags on the nodes to group them semantically. 

* Special monitors for the numerical nodes that give the mean and the standard 
deviation and that have three display modes (normal, relative and relative + 
curve) 

* Possibility to display the delta (probability variation) on the monitor between 
each evidence introduction/removal 

* Possibility to specify costs to each node, i.e. the cost implied for knowing the 
value of the nodes (useful for the adaptive questionnaires) 

* Possibility to set an image background (a map, a logo,...) 
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Tool Name: OCCAM - Organizational and Cultural Criteria for 

Adversary Modeling 

Website: http://www.cra.com 

Company Name: Charles River Analytics, Inc. 

General Features 


General Note: 

The following questionnaire pertains mainly to Bayesian network technology which 
is only one part of our intelligence operations decision-aiding system. In addition, our 
system allows the user to construct and fire rules using a commercial, off-the-shelf rule 
engine and editor. Both the rules and belief networks are able to interact with the various 
entities represented in our tool (e.g., cultural stereotypes, organizations, individuals). In 
addition, our primary display, the social network, allows the user to quick identify 
individuals, groups, and events (and the relationships among these) to run these rules and 
belief networks on. The application level data is stored in a database, mapped onto a 
domain model, and then operated on by rules and/or belief networks in a unified fashion 
via our inference engine user interface. 


1. Ease of use: 

■ Is there feedback for processing delays? For most types of delays, yes (e.g., for rule/BN processing) 

■ Do menu items and text descriptions use language that is commonly understood? Yes 

■ Are menu items in standard places? Are layouts, terms, icon screen positions, windows, dialog boxes consistent? 
Yes 

■ How many steps does it take to perform a frequently used action (desire not buried in menus; not multi-step)? It 
depends on the action - many actions are tied to short-cut keys. Furthermore, some actions are tied to 
multiple user interface elements (e.g., a toolbar button, a short-cut key, and a pull-down menu item) to 
promote both ease-of-use and ease-of-learning. 

■ Do you show user that he/she has made a mistake? How is user aided in correcting mistakes? It depends on both 
where in the system the mistake is made and what type of mistake is made. 

m Do you visually display system objects, actions, and options to users or are these available textually? How do you 
aid users’ memory for these choices? Are instructions easily available if the user forgets or loses his/her way? 
Both toolbars and menus are used to present available actions, objects, preferences, etc. ToolTips are 
available for all toolbar items. 

■ Does your system cater to novice and expert users? Allow users’ to create shortcuts for frequently used actions? 
The system does cater to both novice and expert users (in particular, by providing explanatory interfaces 
and visualizations to the novice, and more sophisticated editing and deductive reasoning tools to the expert). 
We don’t currently allow users to dynamically generate shortcuts, but we provide them ourselves (e.g., ctrl- 
a selects all, ctrl-s saves, etc.) 

■ How is help and/or documentation provided to the user? Tooltips provide additional help on all actions and in 
some displays. Help menu functionality exists, but full technical documentation is still underway. 

■ How does your software handle making best use of limited screen “real estate”? Use of information rich 
displays coupled with extensive use of scrollbars and split-screens maximize screen “real estate”. 

■ Top 5 “trouble” reports? N/A 
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2. Cost: Seat license; upgrades; support; maintenance; (for government customer) 

Proprietary 

3. Maturity of the tool: (Number of years in the market, # versions/upgrades) 

Components of the tool have been under development for 5+ years, other 
components are the result of more recent (past 2 years) research and 
development efforts. 

4. Targeted Industries: 

Proprietary 

5. Major clients: 

Proprietary 

6. Type of Operating system supported: List all applicable OS 

WindowsXP and Windows2000, currently, plans to expand to Unix platforms in 
the future. 

7. Recommended Hardware: (list both minimal and optimal) 

a. Processing power: P4 1.6+ 

b. RAM: 512 MB 

c. Other: 

8. Architecture support: 

a. Client - Server Model: Yes O No ^ 

b. Desktop: Yes 1^ No I I 
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Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes ^ No CH 

b. Decision Networks/Influence Diagrams: Yes I I No 15^ 

c. Both Bayesian Nets and Influence Diagrams: Yes D No ^ 

d. Decision Trees: Yes No ^ 

2. Model Building capability: 

a. Using GUI: Yes lEI No n 

b. Other: Using a custom language, batch mode etc 

c. From databases (i.e. Structure learning): Yes O No ^ 

3. Type of graph support: 

a. Only Undirected graph: Yes Q No ^ 

b. Only Directed graph: Yes 1^ No I I 

c. Chain graphs (i.e. mixed undirected and directed): Yes O No ^ 

4. Conditional Probability Table: 

a. Specify Tables: Yes 1^ No Q 

b. Specify Equations: Yes O No ^ 

(not currently available within the CPT editor, but full cut and paste capability 
to/from Excel is supported) 

c. From databases (learn from training sets): Yes ^ No Q (supports fully 
observable and partially observable data sets) 

d. Other: Various other proprietary user interface techniques. 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes ^ No d] 

b. Cyclic Dependency structure check among variables: Yes ^ No O 

6. Inference Algorithm used: 

Hugin Junction Tree Algorithm (not product, just algorithm) 
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7. Analysis techniques provided/supported: List and explain 

Mutual information sensitivity analysis 

8. For tools using Bayesian Networks (Ref. Col B data) are there built in methods to 
trim the network or otherwise limit computational complexity with “relevance 
reasoning” or any other method: Yes I I No 1^ 

a. If yes, explain: 

Performance and Extensibility 

1. Language used to develop: 

a. GUI: Java 

b. Computational Engine: Java (for both rules and BNs) 

2. Is source code generally available to a government end user: Yes D No ^ 

(If no, is there any circumstance where the source code could be made available?) 

This would have to be negotiated. 

3. Is the API available to the end user: Yes 1^ No I I 

4. Availability of the tool’s performance measurement: Yes O No ^ 

a. If yes, please indicate the benchmark/procedure used to measure performance: 

5. Maximum Nodes a single model can support: (Max nodes a model can support when 
running on the recommended system) 

Only limited by CPU and memory. On the recommended system, on the order of 
10^ (but still depends on network density). 

6. Error recovery support (if something crashes, what happens?): 

Use last saved state (no intermediate or auto-save features currently iuiplemented) 
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Interoperability 


1. Ability to interface with databases: Yes ^ No O (The application supports SQL 
queries to retrieve data used as evidence in the networks. Also, the rule engine 
component can store rules in a database. Bayesian network component receives 
data from the database via the domain model of the main application.) 

a. If Yes, 


i. Using Open Data Base Connectivity (ODBC): Yes ^ No CH 

1. If Yes, list the data bases the ODBC driver supports: the JDBC 
2.0 bridge used supports the ODBC 2.x and ODBC 3.x 
driver manager and drivers, so any database that supports 
those standards 

ii. Databases it can interface directly with(list them): anything 
compatible with the JDBC/ODBC bridge 

b. Does it support SQL interface: Yes ^ No Q 

c. Can models created in the tool be saved to a database: Yes 1^ No I I 

(not Bayesian networks, the application level data is saved in a database 
and any ruleset may also be stored in a database) 


2. Format used to the save the model: xbn (standard xml BN format) for BNs 
proprietary ruleset format for rules 


3. Export the model to other applications: Yes 1^ No I I 

a. If yes, list the applications the model can be exported to: 

Any application that will read the common xbn format. 
Rulesets are not currently exportable 


4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes 1^ No I I 

a. If Yes, list applications that it can be exported to: 

This feature is currently under development: 

BNs: the model can be exported to any application that reads xbn files 
Rules: the model cannot currently be exported 
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Domain model: the model is stored in a database accessible via SQL queries 
Also, results of analysis can be captured in standard image formats (e.g., 
JPEG, GIF) 

5. Tools uses open standards: Yes 1^ No I I 

a. If yes, explain: In the BN model: save model to xbn files (xml bnet 

stadard), cut/paste CPT entries to/from Excel. In the domain model: SQL 
syntax for querying data in domain model 

6. Provide group collaboration on model building and analysis: Yes Q No ^ 
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Tool Name: DXpress Solution Series 
Website: www.kic.com 

Company’s Name: Knowledge Industries, Inc. 

General Features 

1. Cost: To be negotiated - depends upon user domain, degree of exclusivity and 
usage. Some domains are not available for license. 

2. Maturity of the tool: Fully mature software developed since 1992, no changes 
to the Inference Engine since 1999 and only minor editing for the editor and 
testing segments. No outstanding trouble reports, in daily heavy commercial 
usage for web-based interactive diagnostic applications. First sales in 1993. 

3. Targeted Industries: The initial version was designed for medical diagnosis. 
The current software has been developed to be domain independent and has 
been used in aerospace, metals manufacturing, medical clinics, locomotive 
repair, automotive repair, internal auditing and community relations as well 
as proprietary applications.. 

4. Major clients: The Knowledge Industries' client list is proprietary and will be 
disclosed as appropriate. It includes Fortune 100 firms as well as start-up 
companies. 

5. Type of Operating system supported: Microsoft W2000, NT4.0, XP 

6. Recommended Hardware: 

a. Processing power: 500 mhz or greater 

b. RAM: Inference Engine - 1.4 MB, Compilier - 5.7 MB, Test System - 
2.5 MB 

c. Other: Sufficient disk space to store the Bayesian Knowledge 

. Databases. Each Dayes network is compiled by DXpress and is stored 

as an Object (file) that can be executed by either WIN-DX for test and 
validation purposes or operated by the APl/dll that resides within the 
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End User’s application. We refer to these in the collective as the 
Knowledge Databases. 

Can you please explain what you mean by Bayesian Knowledge 
Databases? 

7. Architecture support: 

a. Client - Server Model: Yes 1^ No I I 

b. Desktop: Yes 1^ No I I 

Additional Comments: There are three separate modules. DXpress is the 
Editor/Compilier, Win-DX is the test and validation module and API/dll is the 
Inference Engine that executes the compiled Bayes network. Standard practice is to 
develop and test on the desktop and upload to the Inference Engine running on a 
server. The Inference Engine (API/dll) is state-free. 


Input Manipulation 

1. Type of Network support: 

a. Bayesian Networks; Yes ^ No [I] 

b. Decision Networks/Influence Diagrams: Yes fl No I I 

c. Both Bayesian Nets and Influence Diagrams: Yes dl No dl 

2. Model Building capability: 

a. Using GUI: Yes S No □ 

b. From databases(i.e. Structure Learning): Yes dl No ^ 

c. Other: There is a graphics editor for building the Bayes Network. 

3. Type of graph support: 

a. Only Undirected graph; Yes dl No I I 

b. Only Directed graph: Yes ^ No dl 

c. Chain graphs (i.c. mixed undirected and directed): Yes d] No I I 
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4. Conditional Probability Table: 

a. Specify Tables: Yes No FI 

b. Specify Equations: Yes d] No ^ 

c. From databases (learn from training sets); Yes O No ^ 

d. Other; There currently is no direct method of connecting external sources of 
Conditional Probabilities to the CP tables. Such a feature has been designed 
and can be added if needed. All CP tables entries must be completed prior to 
compilation. Incomplete information is not allowed. 

Is there support for Noisy-Or nodes (if a child node has a large parent set)? 

Yes 

Does DXpress support negative and likelihood evidence? (Like Hugin and 
Netica) 

I’m sorry; I have not examined either of these packages in enough detail to 
respond. I do not know what is meant by negative and likelihood evidence as 
opposed to Conditional Probabilities. I suspect that this is a result of 
differences in nomenclature, not capability. 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes ^ No O 

b. Cyclic Dependency structure check among variables: Yes ^ No O 

6. Inference Algorithm used: Proprietary, to be discussed under conditions of 
confidentiality as needed. 

Is it based on any standard junction tree algorithms that are generally applied to 
Bayesian Nets? Also, does DXpress support both exact and approximate inference 
algorithms? 

A response to this question will be forthcoming. 

7. Analysis techniques provided: Execution of the network may be coded directly 
into the system so that, depending upon the state of an input variable, entire 
segments of the network will be deemed "not applicable" and will not be 
examined - i.e., yes. 
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For this question, I was thinking along the lines of Sensitivity analysis or 
Importance analysis - where the user selects a node (target node) and wants to 
find out how the other nodes in the network impact it. Does DXpress provide such 
support? 

Ah ha! Yes, we have what we call a “debug” function that allows the elicitor 
to identify selected nodes to be “debug” nodes and a complete sensitivity 
analysis can be made during the test and validation phase of BBN 
construction using WIN-DX. 

Where is this debug function? Also, when you mean by sensitivity analysis during 
using WIN-DX, are you referring to how the beliefs in the possible disorders 
changes based on the evidence entered. 

8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes ^ No O 
a. If yes, explain: The technology is proprietary. 

Additional Comments: Value of Information is used. 

Performance and Extensibility 


1. Language used to develop: 
a. GUI: C++ 


b. Computational Engine: C-H- 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user: Yes ^ No O 

4. Availability of the tool’s performance measurement: Yes IXI No FI 
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a. If yes, please indicate the procedure in measuring the performance: Load 
testing. 

I am not clear on what you mean by Load testing? For this question, we want to 
know if you have done any performance measurements of DXpress for some set 
of problems. If so, are the results available? 

A licensee performed load tests prior to selecting our software. In Summary 
form — on a 455 MHz computer using a large Bayes Network (over 200 
nodes) and instantiated with all input simultaneously the test system 
processed over 1,000 transactions per minute with no measurable delay. 

5. Maximum Nodes a single model can support: Performance limit has not been 
established, however, 400-node, multi-fault networks have been built and 
operated. In general, the limitation on node number is the ability of the 
developer(s) to understand the resulting network at a deep level. We have 
found that for a team of experienced elicitors, an effective limit of 400 nodes 
per network applies. We solve the large network problem by a domain 
division method. For example, in one major application of over 700 
networks, the applicable network is selected by a meta-network. 

6. Error recovery support: Yes 1^ No I I 

a. If yes, explain: Not a clear question in this context. Each module has 
operational diagnostics. Do you refer to the Inference Engine? The 
Editor/Compilier? 

We want to find out what kind of error recovery support the tools provides to the 
user when developing the model and running inference on it. Say, if I am 
constructing a model, does the tool have a built in feature to auto-save the model 
every x units of time, so if my system crashes, there is a way to recover the last 
saved version? 

We do not have automatic “save” implemented. The User must save the 
BBN as it is develuped. 
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For API, what kind of Error recovery support is provided? - are the error 
messages well documented? 

We think the error messages from the API are well documented - but none 
of our clients have reported any error messages in the last four years of 
heavy commercial usage of the API. 

When running the inference algorithm, does the tool let the user know if it ran out 
of memory? 

We have never had this problem. Our inference system depends upon a pre¬ 
compiled BBN in contrast to interpretive systems, the API is optimized for a 
small footprint and even the largest of the BBNs that we have tested (over 
400 nodes) will run on a Pentium I computer with limited memory. 

Additional Comments: 


Interoperability 

1. Ability to interface with databases: Yes 1^ No I I 

a. If Yes, 

i. Using ODBC: Yes □ No □ 

ii. Databases it can interface directly with (list them): The potential 
exists, but has not been implimented. 

I have noticed that DXpress does not support structure learning (true) and 
conditional probability learning (true). So, if this featme is implemented, will it 
useful only to save the Bayesian net models to databases directly (we do save our 
compiled networks and they could easily be placed into a database system). 

No, we would allow the importation of Conditional Probability tables from 
external sources to modify an existing BBN. 

b. Does it support SQL interface: Yes I I No 1^ . 

c. Can the models be saved to a database: Yes 1^ No I I 

2. Format used to the save the model: Proprietary. 
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3. Export the model to other applications: Yes I I No 1^ 

a. If yes, list the applications the model can be exported to: All file formats 
are proprietary. 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes ^ No O 

a. If Yes, list applications that it can be exported to: Any 
Can you please list some of the applications? 

We routinely export files from the Test and Validation module (WIN-DX) to 
Excel and other similar programs in a comma-delimited format to allow 
graphic display of the individual Case that is being evaluated. 

5. Tools uses open standards: Yes O No ^ 

a. If yes, explain: 

6. Provide group collaboration on model building and analysis: Yes ^ No Q 

Additional Comments: Has had extensive use by a team building over 700 
networks for automotive diagnosis. 


Additional Notes : Software is industrial-strength, runs all of the time and runs 
well, handles both single fault and multiple fault Bayes networks and is feature 
rich. Well suited for human elicitation and can be readily adapted for 
automation of conditional probabilities from external sources. See 
www.Symptomedix.com for an interactive diagnostic for headaches. Note that 
the sequence of the questions in the demonstration is based upon all prior inputs 
and that the diagnosis becomes stable after the first 8 to 12 of the 150+ inputs 
have been requested. 
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Tool Name: Ergo 

Website: www.noeticsystems.com/ergo 
Company’s Name: Noetic Systems Incorporated 

General Features 


1. Cost: $200 GUI, $100 run-time library; volume discounts are available 

2. Maturity of the tool: Developed and maintained since 1989; no bug reports for 
version 1.0 (GUI or API) for over 1 year. 

3. Targeted Industries: Experts in any field who want to transfer their expertise 
to a computer-based expert system 

4. Major clients: Licensees include Cisco Learning Institute, General Electric, 
Educational Testing Service, dozens of university/academic licensees in 
North America, Asia, and Europe 

5. Type of Operating system supported: GUI: Macintosh, Windows. 

Run-time API: Macintosh, Windows, Unix 

6. Recommended Hardware: 

a. Processing power: No minimum 

b. RAM: 1Mb 

c. Other: Disk space: 1 Mb 

7. Architecture support: 

a. Client - Server Model: Yes I I No 1^ 

b. Desktop: Yes 1^ No I I 

Additional Comments: 


146 







Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks; Yes ^ No HH 

b. Decision Networks/Influence Diagrams: Yes D No ^ 

c. Both Bayesian Nets and Influence Diagrams: Yes D No ^ 

2. Model Building capability: 

a. Using GUI: Yes IE! No □ 

b. From databases(i.e. Structure Learning): Yes lEI No f~l 

c. Other: From scripts (i.e., replay network-construction events) 

How can it do structure learning when there is no support for database interface? 
Replay network-construction events??? Explain 

3. Type of graph support: 

a. Only Undirected graph: Yes fl No FI 

b. Only Directed graph: Yes ^ No CD 

c. Chain graphs (i.e. mixed undirected and directed): Yes Q No Q 

4. Conditional Probability Table: 

a. Specify Tables: Yes lEI No fD 

b. Specify Equations: Yes fP No lEI 

c. From databases (learn from training sets): Yes lEI No f~l 

d. Other: 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes ^ No CD 

b. Cyclic Dependency structure check among variables: Yes ^ No CD 

6. Inference Algorithm used: Proprietary version of Lauritzen-Spiegelhalter; 
user can also enter likelihood ratios as evidence; inference can be performed 
in batch mode for validation, from GUI or API 

7. Analysis techniques provided: Batch processing; log file 
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8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes I I No 1^ 
a. If yes, explain: 

Additional Comments: Client-server architecture and database connectivity are 
under development. 


Performance and Extensibility 

1. Language used to develop: 

a. GULC-H- 

b. Computational Engine: C-H-; API has C or C-h- interface 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user: Yes ^ No Q 

4. Availability of the tool’s performance measurement: Yes I I No I I 

a. If yes, please indicate the procedure in measuring the performance: 

5. Maximum Nodes a single model can support: Limited by available memory; 
maximum number of conditional probabilities per node is 2^32 

6. Error recovery support: Yes ^ No Q 

a. If yes, explain: GUI and API return error codes for all 
errors/wamings 

Additional Comments: 


148 







Interoperability 

1 . Ability to interface with databases: Yes I I No 1^ 

a. If Yes, 

i. Using ODBC: Yes □ No □ 

ii. Databases it can interface directly with (list them): 

b. Does it support SQL interface: Yes I I No 1^ 

c. Can the models be saved to a database: Yes I I No 1^ 

2. Format used to the save the model: Proprietary binary (more efficient); text; 
XML 

3. Export the model to other applications: Yes ^ No Q 

a. If yes, list the applications the model can be exported to: 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes ^ No n 

a. If Yes, list applications that it can be exported to: GUI saves graph to 
clipboard for export to drawing programs 

5. Tools uses open standards: Yes I I No 1^ 

a. If yes, explain: 

6. Provide group collaboration on model building and analysis: Yes I I No 1^ 
Additional Comments: 

Additional Notes ; Inference based on docking has also been implemented. This 
advance supports the decomposition of a network into diagnostic (system) and 
evidence subnetworks; evidence subnetworks are "docked" to the diagnostic 
module when they are instantiated, leading to greatly reduced computational 
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complexity for the diagnostic module, without loss of accuracy as occurs for 
stochastic-inference methods. Please see the following reference for more 
information: 

Almond RG, Herskovits EH, Mislevy RJ, Steinberg LS. Transfer of information 
between system and evidence models. Heckerman D and Whittaker J (Eds.). 
Proceedings of the Seventh International Workshop on Artificial Intelligence 
and Statistics, 1999, pp. 181-186. San Francisco: Morgan Kaufmann. 
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Tool Name: Hugin Developer 
Website: www.hugin.com 
Company’s Name: Hugin Expert A/S 

General Features 

1. Cost: Please see price quote separately 

2. Maturity of the tool: Hugin API - 13 years - Hugin GUI - 10 years 

3. Targeted Industries: Medical industy - this was the starting industry - now as 
Hugin is a general purpose tool, we aim at many different industries. 

4. Major clients: Please see our reference list. Note: No reference list provided. 

5. Type of Operating system supported: Windows, Unix, Linux 

6. Recommended Hardware: 

a. Processing power: 1 GHz 

b. RAM: 256 Mb 

c. Other: 

7. Architecture support: 

a. Client - Server Model: Yes I I No 1^ 

b. Desktop: Yes ^ No D 


Additional Comments: Hugin API compiles with ansi C, and the Hugin GUI 
requires Java. It is Hugins policy to be open for the requirements of the customers. 
If a customers e.g uses another operating system that the mentioned ones, we are 
open for this. It only requires that the customer makes the operating system 
available to Hugin for compiling. 






Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes ^ No O 

b. Decision Networks/Influence Diagrams: Yes ^ No HU 

c. Both Bayesian Nets and Influence Diagrams: Yes ^ No D 

2. Model Building capability: 

a. Using GUI: Yes Kl No □ 

b. From databases(i.e. Structure Learning): Yes ^ No O 

c. Other: 

3. Type of graph support: 

a. Only Undirected graph: Yes ^ No HU 

b. Only Directed graph: Yes ^ No HD 

c. Chain graphs (i.e. mixed undirected and directed): Yes ^ No HU 

4. Conditional Probability Table: 

a. Specify Tables: Yes 1^ No I I 

b. Specify Equations: Yes ^ No HU 

c. From databases (learn from training sets): Yes ^ No HD 

d. Other: Supports missing information 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes ^ No HU 

b. Cyclic Dependency structure check among variables: Yes ^ No HU 

6. Inference Algorithm used: Hugin Propagation - a.k.a junction tree 
propagation 

7. Analysis techniques provided: Conflict analysis 
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8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes I I No 1^ 
a. If yes, explain: 

Additional Comments: 3.a - easily corrupted 

Performance and Extensibility 


1. Language used to develop: 

a. GUI: JAVA 

b. Computational Engine: C 

2. Is source code available to the end user: Yes I I No 1^ 

3. Is the API available to the end user: Yes ^ No O 

4. Availability of the tool’s performance measurement: Yes 1^ No I I 

a. If yes, please indicate the procedure in measuring the performance: 
Please see page 8, 9 and 16 in the attached slides. Note: No slides 
provided. 

5. Maximum Nodes a single model can support: RAM limited only, please 
see page 8 and 9 in the attached slides. Note: No slides provided. 

6. Error recovery support: Yes 1^ No I I 

a. If yes, explain: All functions set error code 


Additional Comments: C-H-,VB,JAVA: Throw/Catch Mechanism 
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Interoperability 


1. Ability to interface with databases: Yes 1^ No I I 

a. If Yes, 

i. Using ODBC: Yes |E1 No □ 

ii. Databases it can interface directly with (list them): Oracle 8 i 

b. Does it support SQL interface: Yes I I No 1^ 

c. Can the models be saved to a database: Yes I I No 1^ 

2. Format used to the save the model: NET, HKB 

3. Export the model to other applications: Yes 1^ No I I 

a. If yes, list the applications the model can be exported to: B-course, Genie, 
Samiam 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes 1^ No I I 

a. If Yes, list applications that it can be exported to: BMP 

5. Tools uses open standards: Yes Q No Q 

a. If yes, explain: All major Algorithms published, ANSI C (API) 

6. Provide group collaboration on model building and analysis: Yes Q No ^ 
Additional Comments: 

Additional Notes ; In general Hugin are open towards special customer needs 
and wishes. This means that if our customers needs any further functionality 
which not are implemented in the standard product, we are willing to consider 
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Tool Name; PSYOP PT 

Website: www.mtisurv.com/psyopdemo 

Company Name: Metrica, Inc. 

General Features 


1. Ease of use: 

■ Is there feedback for processing delays? Yes 

■ Do menu items and text descriptions use language that is commonly understood? Yes 

■ Are menu items in standard places? Are layouts, terms, icon screen positions, windows, dialog boxes 
consistent? Yes 

■ How many steps does it take to perform a frequently used action (desire not buried in menus; not 
multi-step)? 1 or 2 steps 

■ Do you show user that he/she has made a mistake? How is user aided in correcting mistakes? 

■ Do you visually display system objects, actions, and options to users or are these available textually? 
How do you aid users’ memory for these choices? Are instructions easily available if the user forgets 
or loses his/her way? Yes 

■ Does your system cater to novice and expert users? Allow users’ to create shortcuts for frequently used 
actions? Mainly to novice users. 

■ How is help and/or documentation provided to the user? 

■ How does your software handle making best use of limited screen “real estate’’? Scrolling and user 
options to reduce font size. 

■ Top 5 “trouble” reports? 


2. Cost: Seat license; upgrades; support; maintenance; (for government customer) 

3. Maturity of the tool: Number of years in the market, # versions/upgrades. 

4. Targeted Industries: Business, medical, govt.,etc. US Armed Forces 

5. Major clients: US Air Force Air Intelligence Agency 

6. Type of Operating system supported: List all applicable OS 

Server: Windows NT 4.0 or higher; Client: Any machine capable of running 
Internet Explorer 5.0 or higher. 
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7. Recommended Hardware: (list both minimal and optimal) 

a. Processing power: 200Mhz or Higher 

b. RAM: 24 Mb ( dependant on OS ) 

c. Other: Server software required IIS 4.0 or higher 

8. Architecture support: 

a. Client - Server Model: Yes I I No Q Yes. 

b. Desktop: Yes Q No O Yes, if the desktop has Microsoft PWS or IIS 
installed. 


Input Manipulation 

1. Type of Network support: 

a. Bayesian Networks: Yes fl No f"! 

b. Decision Networks/Influence Diagrams: Yes [HI No [HI 

c. Both Bayesian Nets and Influence Diagrams: Yes [H No HI 

d. Decision Trees: Yes Q No O 

2. Model Building capability: 

a. Using GUI: Yes □ No □ Yes 

b. Other: Using a custom language, batch mode etc 

c. From databases (i.e. Structure learning): Yes □ No □ 

3. Type of graph support: 

a. Only Undirected graph: Yes O No O 

b. Only Directed graph: Yes □ No □ 

c. Chain graphs (i.e. mixed undirected and directed): Yes HI No HI 

4. Conditional Probability Table: 

a. Specify Tables: Yes HI No HI 

b. Specify Equations: Yes I I No I I 

c. From databases (learn from training sets): Yes HI No HI 

d. Other: Other Input techniques used 
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5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes □ No □ 

b. Cyclic Dependency structure check among variables: Yes □ No □ 

6. Inference Algorithm used: List all relevant algorithms 

7. Analysis techniques provided/supported: List and explain 

® MAUT as evidenced by SMART, and regression based policy specifying 

8. For tools using Bayesian Networks (Ref Col B data) are there built in methods to 
trim the network or otherwise limit computational complexity with “relevance 
reasoning” or any other method: Yes □ No □ 

a. If yes, explain: 

Performance and Extensibility 

1. Language used to develop: 

a. GUI: Active Server Pages, VBScript 

b. Computational Engine: Active Server Pages, VBScript 

2. Is source code generally available to a government end user: Yes Q No Q Yes 
(If no, is there any circumstance where the source code could be made available?) 

3. Is the API available to the end user: Yes 0 No O No 

4. Availability of the tool’s performance measurement: Yes □ No □ No 

a. If yes, please indicate the benchmark/procedure used to measure performance: 
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5. Maximum Nodes a single model can support: Max nodes a model can support when 
running on the recommended system No hard limits. Only limited by physical 
memory. 

6. Error recovery support (if something crashes, what happens?): Software 
automatically saves inputs as the user works. So, if a system failure occurs, the 
user can pickup exactly where they left off. 

Interoperability 

1. Ability to interface with databases: Yes O No □ Yes 

a. If Yes, 

i. Using Open Data Base Connectivity (ODBC): Yes O No Q Yes 
1. If Yes, list the data bases the ODBC driver supports: 

MS Access 

ii. Databases it can interface directly with(list them): None 

b. Does it support SQL interface: Yes Q No Q Yes 

c. Can models created in the tool be saved to a database: Yes n No O Yes 

2. Format used to the save the model: ??? 

3. Export the model to other applications: Yes Q No Q No (?) 

a. If yes, list the applications the model can be exported to: 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes Q No Q No 

a. If Yes, list applications that it can be exported to: Eg: PPT, Word, JPEG, PDF 
format, PS format, etc 
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5. Tools uses open standards: Yes Q NoQ ??? 

a. If yes, explain: 

6. Provide group collaboration on model building and einalysis: Yes Q No Q Yes 
(e.g., can nodes be added as “sub-models” created by another user?) No 
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Tool Name: Netica 

Website: www.norsys.com 

Company’s Name: Norsys Software Corp. 

General Features 


1. Cost: Single licenses: $585 for GUI, $685 for API. Site licenses available for 
5 times as much. API embedded is $20 to $175 depending on volume. 

2. Maturity of the tool: In development since 1992; sold since 1995 

3. Targeted Industries: Industrial diagnosis, financial risk management, 
enviromental planning, decision analysis, user modeling. 

4. Major clients: Exxon, Boeing, AIG Risk Finance, Lockheed Martin, 

Electricite de France, CIA, MIT, Stanford, Motorola, NASA, SAIC, Siemens, 
Rockwell, US Navy, Northrop Grummon, Raytheon and many others (see 
www.norsys.com/clients.htm) 

5. Type of Operating system supported: Windows 95 to XP, MacOS, and API for 
Linux, Sun Solaris and HP-UX. 

6. Recommended Hardware: 

a. Processing power: Practical with 200 MHz but large models require 
more 

b. RAM: Practical with 128 MB but large models require more 

c. Other: 

7. Architecture support: 

a. Client - Server Model: Yes ^ No Q 

b. Desktop: Yes I^No r~l 

Additional Comments: 
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Input Manipulation 


1. Type of Network support: 

a. Bayesian Networks: Yes ^ No O 

b. Decision Networks/Influence Diagrams: Yes ^ No Q 

c. Both Bayesian Nets and Influence Diagrams: Yes ^ No O 

2. Model Building capability: 

a. Using GUI: Yes lEI No □ 

b. From databases(i.e. Structure Learning): Yes l~l No 1^ 

c. Other: Currently adding structure learning 

3. Type of graph support: 

a. Only Undirected graph: Yes O No HH 

b. Only Directed graph: Yes ^ No O 

c. Chain graphs (i.e. mixed undirected and directed): Yes O No O 

4. Conditional Probability Table: 

a. Specify Tables: Yes ^ No D 

b. Specify Equations; Yes ^ No CH 

c. From databases (learn from training sets): Yes ^ No CH 

d. Other: Sequential updating, and missing data can be handled by the EM 
learning or gradient descent algorithms 

5. Validity of the model: 

a. Consistent Probability check of variables in the net: Yes M No FI 

b. Cyclic Dependency structure check among variables: Yes 13 No [3 

6. Inference Algorithm used: Several, mainly junction tree (aka known as join 
tree or clique tree). Logic sampling. 

7. Analysis techniques provided: Sensitivity to findings (mutual information, 
variance reduction, etc), Testing the net for accuracy with a database of 
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cases, "Processing" sets of cases using the netwoiii, "Summing-ouf' sections of 
the network, noisy-or, noisy-and, noisy-max and noisy-sum nodes 

8. For tools that support Bayesian Networks (Ref. Question 1) are there built in 
methods to trim the network or otherwise limit computational complexity with 
“relevance reasoning” or any other method: Yes No Q 

a. If yes, explain: Sensitivity analysis, "summing out" parts of the network 

Additional Comments: 


Performance and Extensibility 


1. Language used to develop: 
a. GUI: C++ 


b. Computational Engine: C-H- 

2. Is source code available to the end user: Yes ^ No ^ 

3. Is the API available to the end user: Yes 1^ No I I 

4. Availability of the tool’s performance measurement: Yes ^ No Q 

a. If yes, please indicate the procedure in measuring the performance: 

Test for prediction or diagnosis accuracy against a database of cases: E>ror 
rate, logarithmic loss. Brier score, confusion matrix, surprise matrix, ROC 
curves, etc. 

5. Maximum Nodes a single model can support: No limit; Easily handles 1000s if 
the graph structure is simple. 

6. Error recovery support: Yes I I No I I 

a. If yes, explain: 
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Additional Comments: Extensive effort has been put into error checking, and 
recovering well. 


Interoperability 

1. Ability to interface with databases: Yes 1^ No f"! 

a. If Yes, 

i. Using ODBC: Yes lEl No □ 

ii. Databases it can interface directly with (list them): Windows 
ODBC databases, such as MS Access 

b. Does it support SQL interface: Yes 1^ No I I 

c. Can the models be saved to a database: Yes Q No ^ 

2. Format used to the save the model: DNE, NET A, can also read several other 
formats, such as Ergo, DX Express, Hugin. XML implemented but not yet 
available. 

3. Export the model to other applications: Yes Q No O 

a. If yes, list the applications the model can be exported to: Some other 
programs can read Netica format, but we don't have any experience 
or knowledge of how well. 

4. Ability to export results/graphs generated from the “what-if ’ scenarios or the 
analysis: Yes I I No ^ 

a. If Yes, list applications that it can be exported to: Nice graphics of the 
net can be copied and pasted into Microsoft office products. Tables of 
conditional probabilities and inference results can be pasted into 
Excel or a text file (or read from them). 

5. Tools uses open standards: Yes I I No I I 

a. If yes, explain: 
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6. Provide group collaboration on model building and analysis: Yes O No ^ 


, Additional Comments: 
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DARPA’s Wargaming The Asymmetric Environment (WAE) Program 

What is WAE’s mission? The mission of the WAE program is to develop and 
demonstrate models and tools “timed” to specific adversaries, thus enabling analysts and 
decision-makers to better anticipate, predict, and act against those who threaten U.S. and 
Allied interests. Asymmetric adversaries, for WAE’s purpose, span the continuum from 
guerrilla warfare as experienced in the Balkans to the present terrorist threat. The 
resultant technology will yield an automated Continuous Indication and Warning System 
composed of multiple group-specific predictive models (FY 04). 

What are WAE’s research questions? The research questions for WAE are numerous 
and include at a high level; do pre-incident indicators exist that provide relevant 
predictions of our adversaries’ future behavior? If predictive pre-incident indicators 
exist, what predictive fidelity do they possess? Can these pre-incident indicators predict 
the timeframe, target, direction of interest, or tactical characteristics of future attacks? 
What is the shelf life of these pre-incident indicators? Can these pre-incident indicators 
provide any insight into how to influence these adversaries? As quick disclaimer, WAE’s 
technology does not possess the ability to predict the specific day, time, target address, 
and method of an attack. However, WAE does contend that prediction of an adversary’s 
behavior is possible at a detail level that at a minimum dramatically increases the 
specificity of the indication and warning space. The metrics for assessing WAE’s success 
are straightforward; the predictive technology is frequently validated against both 
historical information and real-time information. 

How does WAE’s approach differ from the current analytical approach? WAE’s 
approach is vastly different from the current analytical approach and supporting 
technologies. This difference is best articulated in an analogy of attempting to predict the 
behavior of an electronic circuit of unknown design contained within a black box. The 
current analytical method and tools will attempt to re-create the design of the circuit 
within the black box; understanding the major components and their respective 
interactions. WAE’s approach, on the other hand, concentrates on the environment 
external to the black box, specifically the input and output behaviors. Thus WAE’s 
approach is to derive the behavioral transform function by analyzing the relationship 
between the input and output signals across a range of inputs. The result is a transfer 
function that predicts the circuit behavior without replicating the circuit design. 

Why is this approach important? This approach is important for three reasons. First, 
WAE’s approach provides reliable predictions of near-term future behavior (continuous 
indication & warning) while the existing detection approach and technology, at best, 
provides a description of recent past behavior. Second, WAE’s approach uses a different 
and more readily available information base than current detection approaches and can 
thus derive predictive models when detection approaches falter. An example of this is 
the Greek terrorist organization referenced in the results table above. This group 
operated for 27 years without the US intelligence community knowing any group 
members. Without knowledge of the group members, the US intelligence information 
consisted largely of forensic information. WAE’s approach derived pre-incident 
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indicators that were stable for the early, middle and final years of this group. Third, the 
pre-incident indicators that drive WAE’s predictive models are in form that is consistent 
with and usable by the information operations community, with the added benefit of also 
provided observable metrics. 

What are WAE’s predictive results? WAE has derived models of a number of 
asymmetric groups in conjunction with our operational partners. WAE’s validates its 
predictive models through both historical and real-time tests against the operation target. 
An example of WAE’s initial results is represented by recent tests of a predictive model 
of a specific European Terrorist organization. The model was derived from a training set 
of attacks and non-attacks that spanned the life of this organization. The results are 
presented here as the percentage of correct predictions along two dimensions. First is the 
fidelity of the attack characteristics: attack/no attack, target, direction of interest, and 
tactical characteristics. The second is the predictive accuracy, true positive and true 
negative. True positive, for example, represents the percentage of correct predictions that 
the next attack would reflect the nature of each attack characteristic. Conversely, true 
negative represents the percentage of correct predictions that the next attack would not 
reflect the nature of each attack characteristic. For example, if the model predicts the 
next attack is against a US asset, it is also predicting the next attack will not be against 
the Host, Adjacent, or other International countries. 

As you can see the results are high across the board and in fact, from a statistical 
perspective, the predictive accuracy for each attack characteristic is significant well 
beyond the traditional p<.01 criteria. What is clear, at least for this group, is the 
existence of discemable, predictive patterns to their behavior at a level specific enough to 
support the indication and warning process. The signatures clearly distinguish between 
the environmental conditions preceding an attack versus no attack, a civilian versus a 
military target, a privately owned versus a publicly owned target, and an attack on a U.S. 
versus NATO target. Furthermore, the results indicate that the fixed set of signatures is 
predictive over the life of this group as well as over the evolution of this group’s tactical 
capability. 
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Unclassified - FOUO 


Predictive Results 




► European Terrorist Group I 

■ Historical test on 82 attacks 

• Modeled group behavior. Did not use data 
on specific group members, money or 
logistics 

► Testing 

■ Model derived on a training set of attack & 
non-attack data spanning the life of the 
organization 

■ Model tested on a separate but equivalent 
data set 

► Results 

• Predictions of each response are statistically 
significant 

• Predictions are stable over life of group 

• Predictions are stable over the group's 
tactical evolution 


Unclassified ■ FOUO 


1 Next Attack 


'IVue + 

Ihie - 

Attack 

99% 

93% 

No Attack 

93% 

99% 

Next Target 

Civilian 

100% 

100% 

Military 

100% 

100% 

Private 

89% 

80% 

Public 

80% 

89% 

Next Direction of interest 

U.S. 

86% 

93% 

Host 

93% 

100% 

Adj acent 

75% 

99% 

International 67% 

95% 

Next tactic 

Direct Fire 

78% 

88% 

Indirect Fire 

46% 

82% 

Bombing 

68% 

73% 


Who are WAE’s operational partners? Since its inception (2000), WAE has worked 
cooperatively with DoD and the intelligence community to develop and demonstrate 
predictive models and tools “tuned” to specific adversaries, thus enabling analysts and 
decision-makers to better anticipate, predict, and act against those who threaten U.S. and 
Allied interests. Our partners include the Joint Information Task Force for Counter 
Terrorism (JITF-CT), the Joint Staff J3 - Deputy Director for Information Operations 
(J39), and the Joint Information Operations Center. These partnerships range from 
supporting an on-site WAE team that works closely with analysts to develop and test 
predictive tools to providing independent review of predictive results and technologies. 

As a result, WAE and the operational community have developed technologies that 
derive group specific pre-incident indicators of impending attacks, converted these pre¬ 
incident indicators into predictive models, and then validated the predictive model against 
both historical and real-time information. As a result of these successful tests, operational 
partners such as the JITF-CT are working with WAE to further test and transition the 
predictive models and tools into their analytical process. 
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Letter of Transmittal 


Mr. Glen Nakamoto 
The MITRE Corporation 
202 Burlington Road (Rte. 62) 
Bedford, Massachusetts 09071 


September 1, 2002 


Dear Mr, Nakamoto, 


Please find enclosed the final report of the Market Research Survey for Time Line Analysis 
Tools in response to the research proposal accepted by your office. We have prepared the final 
report in soflcopy in two fonnats for inclusion in your knowledge Portfolio and the MITRF 
IT Advisor as you requested. 


Thank you for the opportunity to conduct this 
behalf of The MITRE Corporation. I am looking 
working with your office on future efforts of 
portfolio. 


research on 
forward to 
the knowledge 


Respectfully, 


John Wigle, G066 
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Exec 14 tiVC Sum mary 


Since Scpicmber I L 2001 ihc needs ol ihe United Slates Intelligence Cornniunily (1C) 
have increased lor lime line analysis to assist with counler-ierrorisni analysis and the 
Asymmetric Order ot Battle (ASOB) problem. MITRE does not have in-depth experience 
with time line analysis tools as evidenced by their lack in the MITRE Analysis Tool Shed 
and other internal analytic tool collections. Currently, the IC is in the process of 
identifying new signatures and meihoils ol delecting and predicting leiTorism. Time line 
analysis is and will be an important tool in combating terrorism and the Asymmetric Order 
of Baltic (ASOB) problem. 

To better understand the capabilities ol time line analysis tools, and how* they support 
the counter-terrorism mission: the MITRE Information Technology Center (G060) 
initiated an internal time line analysis [mI study. This MITRE study was conducted during 
the summer of 2002. The MITRE study team’s first step was to meet with several MITRE 
and U.S. Government intelligence analysis and collect a vSet of functional requirements for 
a time line analysis tool. Using these requirements the study team perlbrmcd a market 
survey of available pnxlucts that may meet some or all of the requirements. The study 
performed a paper compari.son of ihc tools followed by vendor demonstrations and 
requests for demonstration lime line analysis to be added to the MITRE Analyst Tool 
Shed. The lesults of the study are summarized by a scored comparison matrix of the best 
lime line analysis tools cunenily available. 

Based upon the requirements, the MITRE study team’s research, and the information 
provided by the vendors, our recommendation is i2’s Analyst Notebook version 6. It w'as 
the most intuitive, was verified to have had most of the high-level requirements, and had 
helpful dcKumeniation. The i2 staff w as also very helpful and quick to respond to 
questions and support. Unfortiinaielv it was not due out until the end of the 2(X)2 calendar 
year. 

The other highly ranked tool lo examine clo.sely would be VisuaLinks by Visual 
Analytics. And finally, watch Jbr a Windows release of CaseTrak III by Badge 1022 
Software. The tool remained promising, and showed well for a shareware product. 


The MITRE Corporation 
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/. OVERVIEW 
LI Background 

Since Seplemlx:r 11,2(X)1, the analytical needs of the United States Inielligence 
Community (IC) have increased to identify new signatures* and methods of detecting and 
predicting global icrrorism. There are many areas of automated analysis, including entity 
extraction, text mining, categorization, summarization, and search and retrieval^ However 
time line analysis has been largely overlooked. At the time of printing, lime line analysis 
tools remained a weak point among the MITRE capability as evidenced by the MITRE 
Analyst Tool Shed/^ 

After 9/i 1 several customers engaged in operational analysis requested MITRE to 
integrate analytical tools into their environments and requested easy to use t(H)ls^. Among 
the areas requested in automated analysis was time line analysis. Additionally, several 
MITRE analysts and an engineer‘d were interested in various automated looks, including 
time line analysis. It became apparent very quickly that time line analysis was an 
important tool in combating the Asymmetric Order of Battle (ASOB) problem. 
Additionally, we discovered that every analyst hatl a different need and was grappling 
issues with different levels of complexity ranging from simple to robust. Having tools at 
varying degrees of complexity would be cntical to providing the mo.si effective means of 
automated analysis^. 

The Information Technology Center core technology program funded this study. The 
performance of this study was part of the FTC’s Knowledge Portfolio Program (MITRE 
Information Intranet link: http://g()6().mitre.org/neiwork_site/index.html). 

1.2 Objectives 

Through market research and evaluation of time line analysis tools, explore the cuirent 
state of affairs in the lime line analysis market to belter enable MITRE to respond to the 
analytical community, and to provide the analytical community a l>euer range of tools 
from which to select an automated solution. Specifically to: 

• l^ienninc the high-level automated tool rcquiixMnents of analysts 

• Find new lime line analysis kkiIs not listed in the MU RE Analysis Tool Shed 

• Evaluate tools against the requiivments identified by the analysis. 


* Jerry Coglc. ‘TinwJinc Analysis Tools.’* E-mail lo Julie Gravallesc arul Glen Nukumoto 22 March 2(H)2. 

^ Glen Nakamoto’s Transfer Folder - Knowledge Ponlolio <hup //lransicr.iniirc.org/STF40- 

49/42/13742/Transtcr/Knowledge%20Portfolio/Producis/Ti)pic*MjpC'i>llal'H>rationWcbPresomaii<)n/chK/PKM_E.\p<>n- 
195.him> Augu.st 31.2002 

' Analysis Tool Shed home page. <htlp://idia.miijrc.org/i<K>lshcd/indc\.liim> Augu.si 31. 2(M)2. 

* Mark Maybury. “Re: Timeline Analysis Tools.** E-mail to Jerry Coglo. Rod Hi)llund. and John ( iriltith. 01 April 2(KJ2. 
^ Tom Cant)ll (0073), Barry Costa (C30.5I) and Tom McEniec (G075) 

^ Time Line Analysis Market Research and Investigation Pioptwal. CHK>t» tiaicil March 22. 2(H)2. 
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1.3 Approach 

The study team met with and inierxjewed two analysis from The MITRE Corporation^ 
and from the ILS, Govemmeni^. The analysts interviewed repi-esented a variety of analysis 
disciplines w ith over 80 years of combined experience. From these intervjcws the study 
team develo|>ed high-level requirements for lime line analysis tools. 

The inietA iew s conducted were primarily fcx:used on identifying the manual processes 
used that did not incoiporaie compuicrs or other automated processes. This methodology 
was employed so that analysts, w ho are not cuirenlly using automation, had the 
opportunity to express their needs and desires for an ideal computer ap)plication. By 
deriving requirements from the analyst^ pers[)ective, the study team hopes to achieve 
acceptance from all analysis in the use of the tools evaluated meeting the stated 
requirements. 

The evaluation ot t<K)]s was based on the high level requirements identified in the next 
session. We developed an Excel spreadsheet detailing the high level requirements, whai 
we looked lor within that high level requirement, and indicated for each tool whether it 
met the lequirements or not. Ijie vendors were contacted and given a copy of the high 
level requirements and asked to comment on them, and pmvide a demo or evaluation copy 
of their tool. The study team made a concerted effort to confirm the vendor’s rcmaiks on 
their tool’s ability to satisfy the stated requirements through direct demonstration of the 
tool’s capability. 

In the event a vendor did not respond to our inquiries, the study team did their best to 
evaluate the demo without the vendor's assi.stance. And if the study team was unable to 
verify the capability through a demo or evaluation copy, then the vendor’s remarks were 
relied upon as truthful but were notated that they were unconfirmed. If the vendor neither 
responded nor provided a diieci demonstration of the capability to the team, then the u>ol 
is considered not evaluated for this i*epoii, 

1.4 High Level Requirements 

The vci*y basic requirement for this research effon was a software utility or tool that 
was able to produce a time line w iih asscK'ialed information to allow the user to conduct 
meaningful analysis, assessments and connections. The study team used the expounded 
requirements as a means of creating the Tmils Comparison Chart. The high level 
requirements for a time line analysis lool provided by the intelligence analysts weie: 


Table 1 - High Level Requirements 


1 

Automation of input from various sources and in various forrruits. 

2 

Save the customized method of data import for reoccurring formats. 1 

3 

Save bibliographical data relerences, kind ot events, dates and times for events and 1 


^ Canoll. Tom (CH)7.^) and McEnlcc. lorn (G()7.S). Personal inierviews. 5 April 2002 and 26 April 2(X)2. respectively. 

Unidcmiflcd imelligcncc analyst oHhc US Cuncnuncni Personal interview. 30 April 2002. And an unidcniiticd. Special 
Ageni with the US Ocpanmcni of AgricuUiiiv. ( nniinal Invesiigation Unit. Personal interview. 14 May 2002. 
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:- "1 

L.- 

entities. 

1 

4 

Comment on an event or entity, analogini.s lo a po.si-ii^ noie. i 

5 

Query events and entities by time. dale, reference.s, and kind of event. i 

6 

Place events on multiple threads. 

7 

Gantt chart known problems lo reverse engineer events. 

8 

Move events around time line. 

9 

Expand and/or compress lime line. 

10 

Categorize events with icons. 

1 

11 

Export to various formats, including power point. MS word. Visio, web, PDF 

12 

Output to large plotter devices 

13 

Has an intuitive and user-friendly interface 


1.5 Meanings Used for High-Level Requirements 

Automation of input from various sources was divided into three areas, manual, semi¬ 
automatic and dynamic. Manual meant the user selected the appropriate menu items to 
initiate a manual process of imponing data from a source into the tcxil. Semi-automatic 
meant the tool had a wizard-like interface that assisted the user with importing data. 
Dynamic meant there existed a facility to connect to other data sources similar to ODBC 
or XML, etc. 

Saving customized data import formats was refened to as a filler. A filler was defined 
as a tool’s ability to remember the data format. Similar to an XML DTD, or an ODBC 
driver, the tool could understand the data being imponed once an appropriate filter was 
selected. The Department of Defense (DoD) Regulations*^ requires all DoD intelligence 
agencies to have the ability to understand SGML, HTML, and XML for data exchange and 
interoperability. If a tool was ODBC or XML enabled, it was presumed to have met the 
first two high-level requirements. 

Saving bibliographical data references, kind of events, dates and times for events and 
entities was broken down into bibliographical, time, date, and references. The 
bibliographical data meant the analyst would be able to sec the evidence or citation to raw 
intelligence establishing the event or entity. The kind of event meant the user w'ould be 
able to make a notation of the type of event, like a drug inuisaclion, wire transfer, phone 
call, etc. and is named references in the second column of the Tool Comparison Chart. 
Time and date are self explanatory, except that it was impoHant for analysis to be able to 
leave dates blank if they did know when an event cK'currcd. 

The Post-it® note requirement was liberally construed to include tools that had the 
ability to place a text box next to an event, or place a comment or note in the event itself. 

Querying events and entities by lime, dale, references and kind of event meant the tool 
had the ability to sort, select or only list events or only list entities matching the search 
criteria. The meaning of references for this high-level requirement was semantically 

DoD Joint Technical Architecture version 4. dated 17 July 21K)2. 2.2.2.1.4.1. •Document Interchanjre’* -rhitp /Zwww- 

jia.i tsi .disa. mi l/jia/JTA4(M)71702. pd IS 
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diflereni from high-level reqinirnieni luimher 3. lit this requirement, references meant 
bibliographical reiercnces The search criteria had to allow the user to search by the tour 
areas delinecl. whether the hk) 1 used a son, a select or a listing. The four areas defined 
were time, date, references, and type' ol event. 

Placing events on multiple threads meant the tool had the ability to display several 
time lines or parallel activities on the same time domain or display. 

The ability to Gantt chart known problems to reverse engineer events meant the tool 
had the ability to place events intt^ an alternate view showing tasks or ihre;ids, schedules or 
timelines and their dependencies like a (iantt chart. 

Moving events around the time line meant a user could directly manipulate an event to 
either cxpand/c(impress the time line on each side the event, or that the event could be 
moved venically along a horizontal lime line to facilitate spatial or other notation for the 
analyst. 

To expand and/or compress time line mciint the user could universally define several 
different scales along the same lime line. 

Categorize events with icons meant that the user had the ability to select and assign 
meaningful icons for events and entities. Icons were liberally construed to include 
photographs in JPEG and GIF lomiai or clip art in addition to traditional icons. 

Exporting to various formats, including Micrtisofi Power Point, Microsoft Word. 

Visio, web, and PDF meant precisely that except web meant any of the formats commonly 
used on the Internet to exchange documents and graphics. The formats for the web 
.specifically included, HTML. JPEG, GIF, and text. 

To ensure coverage by DoD and non-DoD agencies and components, tools were 
luriher examined to see if they could read from major data sources like Oracle, Microsoft 
Excel, Syba.se, Microsoft Access. Informix, Ingress, and Microsoft SQL Server*^ 

Output to large plotter devices meant that the tool was capable of working with larger 
printer devices like a D-size plotter with the appropriate wdndows drivers installed on the 
system, or at least was able to tile the chart on regular printer ,sizc paper so it could lx? 
assembled into a large chart. 

Had an intuitive and u.ser-friendly interface was a subjective requirement, but the study 
team did its Ixrsi to be objective. User friendly and intuitive meant that the Uk>1 did not 
require the u.ser to open the help files and manuals loo often. Specifically, if a tool was 
determined not to he intuitive then it meant the tool did not follow the Microsoft Windows 
graphical u.sei inieiface standard.^ or had significant problems preventing the user from 
completing la.sks. 


Joint Tct'hnicul Aahiuaturc Lisi ol Mimd;ilc<J and LnKTiiin^* SiufKiardh. dated 17 July 2(X)2 <htlp7/www-jia.ilsj.di.sa.mil/> 
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1.6 Scoring 

Vendors provided intbrmaiion on the high level requirements, and study team 
members evaluated the tools individually and ni)ies were compared and discussed. There 
w'ere three scorns associated for each tool. They were the requirements score, the 
confidence score and the overall score. Each score is normalized between 0 and 10 with 
zero meaning very poor and 10 meaning outstanding. The requirement score was 
computed by adding up the total number of defined requirements satisfied, then dividing it 
by the total number of defined requirements. The total number of defined requirements 
was twenty-six (26). Then multiplying it by 10 for a normalized .score. 

The confidence score ranked the study team’s level of confidence in the information 
they had about the tool. If the study team was unable to verify if a requirement was 
satisfied, then a “not reported” is shown for the defined requirement on the 1 ch)! 
Comparison Chart. However, if the vendor assened they met the requirement, but the 
study team waslinable to verify the fact, then a '*ves^'‘ or “no^” is listed with a caveat that 
it was considered satisfied only based upon the vendor's statements without verification 
by the study team. The confidence score is computed by totaling the number of defined 
requirements not verified by the study team, then subtracting that number from the total 
number of defined requirements and multiplying the result by 10 lo normalize the score. 

Two products, VisuaLinks and TimeMap. had the vendor statements listed. The study 
team inquired late with the makers of VisuaLinks about the high-level requirements and a 
request for an evaluation copy. There was siiflicienl lime to conduct a cui*sory review of 
their tool, but the vendor prevented the study team from beginning an evaluation. The 
vendor had a very iterative customer .service process in obtaining the answers to the 
requirements, then for permission lo download an evaluation copy, then to obtain a 
password, and finally a server demo licen.se key^’. TimeMap was revisited following 
feedback on the final report and vendor statements were added to the matrix. 

The overall score is simply computed by adding the lequiremenis scoie with the 
confidence score, and again was normalized by dividing by 2. This system of scoring is far 
from perfect but provides a reasonable way to compare the tools w ith one another and this 
type of method closely matched the study team s expert judgment'^. 


Lynee We.sion, Visual Analytics. “RE: New Client Profile lor iniire.org” E-mail lo John Wigle. 29 August 2002. 

Bob Wiss. Cascsol't Software. Icicphonc Interview. 8 Octobi‘r 2(H12 

Dr Lehner (G06A) Meeting on expert judgment tor Natiiinal Inielligenco Priorities Fmmework 29 July 2002. 
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11. KVALIUTIONS 

2.1 Tools Investi^aled 

The were found by using ihc MITRE Analyst Tool Shed and search engines over 
the Inteniei. The tools evaluated were; 

• Time Map by Case Soft 

• WehTAS by Air Force Research Labs, Rome, NY 

• Wat\(yn by Xanalys 

• Orion VIA by Orion ScientifK System^ 

• Analyst Norehook by i2 Inc 

• Pen-Link Analysis Sofm^are by Pen Link 

• CaseTrak III by 1022 Badge Sob w are 

• VisuaLinks by Visual Analytics 

• 

2.2 Discussion on Evaluated Ttmis 

A total ol eight tennis were evaluated for the market survey and research project. Tools 
appear in the same order as the Tool Comparison Chart. The discussion covers the tool’s 
rank, strengths, weaknesses, customer suppon and a recommendation. Unique attributes or 
special considerations of the tend were included for completeness. 

2.2.1 TitneMap 

TimeMap developed by Case Soft sells for an affordable $199 per desktop. It was 
designed to work with companion products CaseMap and TextMap used by law firms, 
private practice attorneys, and prosecutors to prepare their cases for litigation in court. 
CaseSofi had also .sold site licenses to the Department of Justice and Department of 
Defense Criminal Inve.stigaiion communities TimeMap was a simple way to make a 
timeline out of known events’^. It scored an overall 7.7 in a place tie with the Watson 
tool. It was above average on meeting requirements with a score of 6.5. It was easy to 
learn, can be .self taught or taught by demo over the telephone^^. It was one of the few* 
tools that had a Gantt chart and plotter capability, along with better than average ability to 
export to other formats. TimeMap had a semiautomatic method for imtx>rting data from 
CaseMap, which in-turn could semi-aulomatically populate from TextMap^^. It lacked an 
ability to query against the time line. It could not import data from other sources 
automatically for populating a timeline hut simply makes one out of events manually 
inputtedExpect good customer support. They were quick to respond, had 
demonstration software on their website, and were helpful in answering any questions. 
Recommended for analytical shops looking for modest capability at a modest price. 


Thoma.s McLmee “Re: Time Line Analysi.^ Kepon . ’* E-nuiil to John Wigic. 29 September 2002 
*•' Jeffrey Lissiin. Aflomcy. "Evidence Analysis Timcliiu^s Are Not Just for Trial Anymore"' Legal Tech April 2002 Vol. 20, 
No I.Pg. 3. 

Jennifer Wch.sici “added info abom cuvesoU. ** Email to John W^iglc. 29 August 2002. 

Thoma.s McEriicc. “Re. Time Line Analysis Repon...’' E>mail to John Wigle. 29 September 2002 
**' Thonia.s McEnicc. '('a.scSofi Legal Sortuarc Applkuiion^”. E niui! to John Wigle. 17 May 2002. 
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2.2.2 WebTAS 

WebTAS developed by Air Force Research Lalis (AFRL) v^as customized rapid 
development software. It was designed for governniboi or large enteiprise customers. It 
scored an overall 7.9 as the 2^ place tool. It was belter than average on meeting 
requirements at a score of 6.2. It was one of the few tools that had a Gantt chart and could 
dynamically access other data sources, WebTAS was looked at heavily as an impon 
system that automatically put events on time lines. There were some powerful capabilities, 
such as geographic mapping, that were not displayed by any commercial products. The 
events were highly searchable, but the learning cuive was high to retrieve information. A 
two-day specialized training session was rcquircti at $L535 just to begin working with this 
tool*^. The manual was overly specific in some areas and vague in others“‘\ The tool itself 
was not intuitive to use^*. Its wetnesses were definitely an unfriendly interface, a lack of 
ability to export to other formats, and less annoying but simple features like placing 
comments or notes on events^, and moving the events around on the time line. The study 
team and others at MITRE were unable to get WebTAS working^^. and relied on Janet 
Hitzeman*s work with WebTAS to substantiate .some of its capabilities’^. Customer 
support from AFRL is weak. AFRL were no-shows at or cancelled several meetings with 
potential customers before^. It would be e.sseniia] with this product to pay for and ensure 
software development, maintenance and user training weie provided since tliis software 
would be built uniquely for each customer base. Recommend only for large shops with 
political clout and a large budget to support .software development, maintenance and user 
training and to put pressure on AFRL if they fell behind in their support. 

2.23 Watson 

Watson developed by Xanalys was a commercial solution with an undisclosed price 
tag. It was designed for law enforcement, intelligence and government customers. It 
scored an overall 7.7 placing it in a 3”* place with TimeMap. It was above average on 
meeting requirements with a score of 6.5. It was strong on user features and export 
formats. News reports claim it was the world's best analytical tool, and it was voted the 
best intelligence analysis tool by an international law enforcement a.ssociaiion in 1999^^. 
However, it did not support dynamic access to other data sources or filters to simplify 
manual data importing. Expect less than average customer support for tliis tool. There was 
no response to inquiries for pricing information and some requirements. Recommended 
for small to medium shops that can afford to pay for premium customer support if offered. 


Jay Jesse. “Re: WebTAS.” E-mail to John Wigic. 09 February 2(X)2. 

** Janet Hitzeman. “Re: webTAS.” E-mail to Patrick Jones. 11 April 2002. and see next fool mae. 

Janet Hkzeman. “FW: WebTAS” £>maii to Michael Maskalcris. 03 May 2(X)2. 

“ Janet Hitzeman. “FW: WebTAS” E-mail to Michael Maskaleri.s. 03 May 2(X)2. 

“ Barry Costa. “Re: WebTAS.” E-mail to Michael Maskaleris and John Wigle. 06 May 2002. 

John Griffith. “Re: Timeline Analysis Tools.” E-mail to Mark May’bury. Jerry CogJe. Rod Holland. Janet Hitzeman, 
Michael Meridcth. Julie Gravallese, Aaron Lesser. John Wigle and Glen Nakumoio. 01 April 2002. 

“ Herbert Mucks, WebTAS PM “RE: webtas.” E-mail to John Wigic. 14 March 2002. 

“ PRLinc - England. “Global Graphics Announces a Euro 5.1 Million Canadian Government Software Contract for Xanalys 
Inc.” 2 November 2000. 
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2.2.4 Orion VIA 

Orion VIA developed by Orion Scientific Systems offered this commercial solution 
originally developed in cooperation with the Defense Advance Research Projects Agency. 
It was designed as an automated intelligence analysis system, and W'as professed to be a 
fixture in the 1C. It scored an overall 7.1 placing it in 5**' place. It was average in meeting 
requirements with a score of 5.0. The tool offered a Gantt chart and large plotter 
capability, and dynamic import from other data sources. Its shortcomings were weak quer>' 
capability against the lime line, pinir data export formal, and no ability to save diita filters 
to facilitate common data impons to the tool. Expect the average customer support from a 
commercial software vendor. Recommended for .shops to review, but only use if it solves 
unique needs other tools do not address. 

2.2.5 Analyst Notebook 6 

Analyst Noteb<x)k 6 by i2 Incorporated was due out at the end of 2002, and was a 
commercial solution with S3388 price tag per license that includes 1 year of technical 
support. It was developed for the British IC, and had customers all over the world, 
including the United States*^. We got a sneak peek look at the pre-release"^. It scored an 
overall 9,0 taking first place by a signiticani margin. It was outstanding in meeting 
requirements with a score of 8.1. one of the highest on the evaluation. The tool offered a 
rich user feature set. rich data import capability, and an easy to use interface. It could 
import data or connect directly to third party databases"^ like Microsoft Access and Sybase 
through its 1-Bridge product. Events were searchable, and useable for both lime line 
analysis and link analysis Icons were u.sed on the lime line and could be changed or 
customized. 1'he manual contained help documentation that was very good for learning 
how to use the tool. Additionally, Analyst Notebook 6 had a robust Application 
F^rogramming Interface (API) to allow' customization of the tool if needed. Its weaknesses 
were a lack of a Gantt chan capability and a marginal export capability. Expect 
outstanding customer sup|)on from i2. It is no coincidence that Analyst Notebook scored a 
perfect score for confidence. The i2 team was leadily accessible, ensured questions were 
answered, and were very patient with the study team’s questions and needs. 

Recommended for any size shop that can afford the price lag and wants to gel rolling 
quickly with their analytical effoiis, and particularly valuable if the shop has access to 
various data sources. 

2.2.6 Pen-lank Analyst 

Pen-lJnk Analyst by Pen-Link LI D w'as a shareware and commercial prcxiucl 
developetl primtu-ily for law cnforcemeni working with warrant access to telephone 
conversations. The tool places phone calls on a time line to assist law enforcement officers 
investigating and building their ca.ses. It scored an overall 4.2 placing it in last place. It 


PR Newswia*.' i2 Inc W'ins $2M Federal ConirucM tmm FBI” 31 July 2001. 

i2 Analyst Noiehook ft. Pre-rclcasc Sympnsiuni The Key Bridge Marrioi. Rosslyn, Virginia. 7 August 2002. 

^ PR Ncwswirc. “Wai on Drugs. Tcmm.sm to Bcnollt imin i2 Inc.’s Groundbreaking Telephone Call Paiicm Analysi.s 
Software.” Financial News section. 13 March 2(X)2. 
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was below average on meeting reqinremcnls wiih a score of 3.8. The lool was rich in user 
features, including a PDA version of the lool. The only weaknesses known was the lack of 
a Gantt chan capability. The lool ranked second lo Iasi on the confidence scale, and it w^as 
unclear if it supported a rich data import/expon capability, Post-it^ note capability, 
multiple threads, ability to manipulate events on the timeline, and the ability to have 
several scales on the same lime line. Customer support is unknown as well, except it was 
not available on the w'eekend. Recommend Ibr lurtlter consideration by shops that are law 
enforcement oriented. 

2.2.7 CaseTrak III 

CascTrak III developed by Badge 1022 Si'iftw are as sharew'are. It w as designed for the 
law enforcement community users on a tight budget. It scored an overall 0.3 placing it. 
surprisingly, above two other commercial pri>ducis it was below average on meeting 
requirements with a score of 3.1. Its strength w as in data references and query capabilities. 
Its weaknesses were that it was still a DOS based program, had poor data iniport/export 
capabilities, and no graphical interface. Customer support would be problematic for Badge 
1022 Software, which was one person. However, w-e did receive customer support on a 
holiday weekcnd^^l Not recommended at this time except for an (Kcasional analyst who 
might find ii useful. Badge 1022 Sofiwaie is considering redevelopmeni on Windows^’. If 
a Windows version is released it is recommended analytical shops revisit this practical 
tool. 

2JI.8 VisuaLinks 

VisuaLinks developed by Visual Analylics was offered as a commercial product for 
an undisclosed price tag. The vendor supports ihe law enforcement, intelligence^’, 
corporale^*^ and government^ communities. The um)I scored an overall 5.4 placing it 
second to last. However, it scored a 9.2 on its vendor-declared ability to meet 
requirements. VisuaLinks was the link analysis and visualization .software used for Joint 
Intelligence Virtual Architecture at the Delensc Intelligence Agency'*^. The low overall 
score was a result of the study team’s inability lo evaluate the lool. Therefore, the 
confidence score was 1.5. The study team w'as able lo verify from news sources that it did 
have an XML interface and ability to pull data dynamically^^’. Expect frustrating customer 
support tor this tool. Although we inquired late with Visual Analytics, there was sufficient 
time to handle our request lo evaluate the toed ami respond to the requirements^^. We did 


^‘Uiry Rile. Badge 1022 Software. "Re; CascTrak III" E-mail lo John Wigic. .^1 August 2(K)2. 

Sec previous loomoie. 

PR Newswirc. "Visual.ink.s Chosen as Link Analyst.^ Tik>I lor Major L)lA and DISA proiccis**. rinancial .New.s 25 June 
2002 . 

” PR Newswirc. "Visual Analytics Expands International Base into Asia.'*. F'inanciul New.s .section 5 l>ccemher 2001. 

PR Newswirc. "Visual Analytics Inc Licenses Visual Data Mining SoUuare lor tl.se in Counter-Drug Operations", 
Financial News section. 8 November 2000. 

^ PR Newswirc. “VisuaLinks Selected as Best-of-Breed Sysiem foi Liiigc-Scalc U.S. intelligence Pnrgram" F'inunciui New.s 
section. 14 February 2002. 

^ Kyle Bulluck. "F^'cds lo Use VAI App in Drug War" Newsbvtes H November 2(KX» 

John Wiglc ‘Timeline analy.siscapahililic.s..."E-mail to Lynoo Wcsion 21 Augusi 2tK)2. 
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receive a response lo the recjuiremenis'*^. hut we were unable to gel the appropriate license 
keys and login intormalioiv^* lo begin oiii evaluation of the tool within a week. 
Recommend analytical shops ol any size consider this tool based on its rcquirenicnl rating, 
but carefully test drive the kh)! and the customer support before investing your business 
processes in the tool. 


^ Lynce Wc.sion. Visual Analyiic.s. **FW': Tiniolinc analv>is capabilities...** E-mail lo John Wigic. 26 Augu.si 2002. 

L>'nce W'cstnn. Visual Aualylics. "RE: Ncu C lient nmlilc t\>r jwiplcC^mitre.org” E-mai! to John Wiglc 29 August 2002. 
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Table 2 - I'ool Cotiiparisoii (^hart - Part I oC 2 
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Table 2 - T<m)I Comparison Chari - Part 2 <>1*2 
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2.3 Tools Not Investigated 

• File List by Forensics, Inc 

• Show File by Forensics, liic 

• Timi: wizard by WIZdom Analysis Tools 

• Flow W IZahd by WIZdom Analysis Tooi.s 

• Expi*neuceWare^^ by PRAJA. Inc. 

• EgressPro by SimCo Consulting: 

• WinForce by winForce Technologies 

• C-lnsight by MetaEdge Corporation 

• DB2 Intelligence Miner by IBM Corporation 

These tools were briefly reviewed because the vendor did not respond to inquiries, or 
the study team concluded their purpose was disparate from intelligence analysis, or both. 
However, the study team felt it was important for a minor sidebar discussion about these 
tools because they had displayed capabilities in dealing with events referenced by lime and 
date, and the study team freely admitted their limited ability to understand the absolute 
needs of intelligence analysis. Therefore, these tools were included in this report to help 
those who may be interested in letu'ning more about them. 


The MITRE Corporation 


12 


186 








































-^August 31, 2002 


2.3.1 FileList and ShowFile 

FiJeList and ShowFile were products ol I-orcnsics. Jnc. They created a lime line 
analysis of computer file dates and times from one or more computer hard disk drives and 
floppy diskettes. Time lines could be created based on tile access dales, file creation dates, 
file modification dates, and activity associated v\ ith deleting files. Identification of 
computer activity on weekends and outside of normal working hours could be put into a 
separate time line. ShowFile sorts, analyzes, and \ lews databa.se output created by 
FileList. Time line analysis of computer data would be extremely helpful in investigations 
and computer security reviews. 

2.3.2 Flow Wizard and Time WlZard 

The WlZdom Analysis Tool Suite had two useful ttxils called the Flow- WlZard and 
Time WlZard tools. Flow WlZard generated process fiow models lor process mapping. 
Time WlZard was a project management tool utilizing time lines. It would lie u.scd to 
reverse engineer a major development project using a Gantt chart process which was one 
of the high-level requirements identified. 

2.3.3 ExperienceWare 

ExperienceWare**^ was a rapidly deployable business activity monitoring solution 
from PRAJA. Inc‘*^\ Their customers included Zurich Financial Services, FO^X. University 
of Chicago, CBS, Yahoo, General Motors, and Heniy Ford Health System. The tool 
allowed companies to monitor busine.ss processes by plotting activity three dimensionally 
using location, process, and time. One could see that this could be used as a time line 
analysis tool, but the tool would probably be used irregularly from its design and that 
could become problematic. 


2.3.4 EgressPro 

EgressPro was a fire engineering simulator developed by SimCo Consulting, and was 
sold for $1.50 fxjr desktop. The simulator develops time lines and movements of e.scape 
from fires based on established fire protection engineering theory^. Was designed to assist 
architects and fire protection engineers in developing models to predict egress limes for a 
given room, corridor or stairwell^*. The tool runs on a Windows 95/98 or higher operating 
system. The tool has no direct application to intelligence analysis that the study team was 
aware of, but tor the price it may be woilhy of looking at if your analysis includes times to 
egress from locations that are not necessiirily on lire. 


ExperienceWare''* Advcriisemeni. iniemet. 31 August 2(X)2. 

Computer Models for Fire and Smoke . Society of Fire Pn)ieviii»ii EnuiniH'r.s IniemuiiiMtal Surv ey of Computer Models for 
Fire and Smoke. <hltp://www.nremodelsurvey.com/pdf/]igrcs.sPin_2<K)l.pdr> 27 July 2tK)2. 
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2.3.5 winForce 

winForce is a commercial lool ot lereii hy \\ inForce Technologies of Scollsdale, Arizona 
for S595 per deskiop*^^ li was a full leaiurcd case management system designed for 
lawyers and prosecutors. The tool mcliidcil the ability to place data entered into the system 
automatically onto a timeline to present infonnation abi'iut the criminal offense, the 
investigation and the evidence^ . The vendor was unresponsive to the study team’s request 
for information. Their website is hiip:/Avww.bliieknighttech.com/ and includes a 
download section, but a trial version had to be mailed out. 

2.3.6 C-lnsight 

C-Insight a commercial tool ol lered hy MetaEdge Corporation of Silicon Valley is 
designed for customer intelligence lor e commerce business. C-Insight allows businesses 
to review customer traffic in u lime line fashion.**^ 

2.3.7 DR2 Intelligent Miner 

DB2 Intelligent Miner offered by IBM includes lime-sequencing functions. Although 
not a stand-alone lool (requires Business Intelligence Platform'’^), it could be incorporated 
into existing customer solutions. In a news anicle, a biopharmaceuiical company used 
DB2 Intelligent Miner to determine the progressions of diseases with limeline-S**^. 


‘’Welc(»rnc to winRircc TechnoIojiieN’’. winFoav H(»nic Pauc <hup;//www.bIuekniphi!cch.coia/> 31 July 2002. 

Michael Rogers. AUtirncy. “W'inForccr PokIuci Review*'. Le gal Tech . November 2001. Vol. 19. No, 8. pg. 6 
^ PR Newswirc. “MelaEdgc's C-ln.sighi Provides Siaic-Or-7‘hc-Ar! Customer Intelligence for E-Enterprises’* 12 September 
2000, Financial News section. 

*‘IBM Software : Database and Daui Managenvoi ; litteiligent Miner Family t Overview” DB2 Intelligent Miner web page. 
<h!ip://www-3.ibni.coni/sofiware/dala/iiiiiner/> 17 Augu.si 2(K)2. 

^ PR Newswiivv ‘'AxCUill Bio.se ienc’e.s Explores Pathways to disease with IBM Database Technology” 16 August 2002. 
Financial News .scclion. 
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III.SUMMARY 

3.1 Recommendations 

BavSed upon the given requirements, the team's research, and the information provided 
by the vendors, our recommendation was i2's Analyst Notebook Version 6. It was the 
most intuitive, was verified to have had most ol the high-level requirements, and had 
helpful dix:iimentation. The i2 staff was alM'^ very hclptiil and quick to respond questions 
and support. Unfortunately it was not due out until the end of the 2002 calendar year. 

The other tool to examine closely would be VisuaLinks by Visual Anal>aics. 

And finally, watch for a Windows release of CaseTrak III by Badge 1022 Softw^are. 
The tool remained promising, and showed well for a shareware product 

3.2 Conclusion 

There were some powerful tools available Ibi use by the IC on the commercial market, 
but no holy giail.^^ None of them met all the high-level requirements identified by the 
intelligence ana]y.st.s. Of the evaluated lords, a few met most of the high-level 
requirements for time line analysis, and most of the tools met at least half the high-level 
requirements. Tliis disparity of capabilities illustrated the lime line analysis market was 
still maturating. There were still too few' tools available. The need for better time line 
analysis tools still existed for increasing the MITRE Analyst Tool Shed product line and to 
combat the Asymmetric Order of Battle (A.SOB) problem. 

It is possible that the time line analysis market, and in general the automated 
intelligence analy.sis markets, will continue to evolve into a richer market of tools. Newer 
products could be introduced and fuilher dc\eh)pment of existing tools w ill only improve 
the availability and selection of lime line analysis tools. Identification of analytical 
requirements of a customer will remain essential to determining which tool will work best 
for a given envimnmenl. And it may become necessary to customize existing commercial 
software through application program interfaces to .satisfy the needs of intelligence 
analysts. 

This market survey may serve as a baseline for future effons at analyzing the lime line 
analysis market. In the event that the time line analysis market does not grow\ it may be 
practical for MITRE to develop customized API plug-ins for commercial sofiw'are or to 
develop a tool to match the requirements of the intelligence analyst community. 


3.3 Contact Information 

You may direct your questions to the study team at the following phone numbers and 
email addresse.s listed in the table below^: 

Table 3 - Study Team Contact Information 


Jim Bumetti. ‘Timeline Analysis Tools Evaluations. ' E-in;ui i<> John VVielc. 31 May 2tX)2. 
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Senior Advisor Jern Co^lc 

Lead Researcher John XN’igle 

Researcher Jennifer Websier 


(703) 883-6277 
(703)883-1277 
(703) 883-6(339 


Jc()gle(g'iTiitre.()rg 

jwigJe@initre.org 

jenw@niitre.org 
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IV. Glossary of Terms and Phrases 

API - An acronym for Application Program Inteifnce. An iniei face tJe^igned into compute! programs that allow 
third parties to develop external software that works with the application like if it was built into the application 
itself. Typically, the software mt^ules designed to work with applications through an API is called u plug-in. 
because it is imagined to plug into the tool. See Plug-in. 

ASOB - An acronym for ASymmciric Order of Battle. 

Asymmetric Order of Battle - The steps an cntii> gi>es through to read) itself for an irregular tasymmetric; 
offensive or defensive pt^sition. Tlic acronym for this phrase is ASOB Compare with Order of Battle and 
Signature. 

OTD — An acronym for Data 1 able Description. A DTD is a di>c'uincni used U) de.scrii>e the data elements in u 
data file, so a third party will understand how tlic data is organized and what data is stored 

Gcuitt Chart - A specific type of chart used in program management that is named after an American scientist 
who invented it. TTie chart uses milestones to denote the beginning and ending of a sfxjcific ta.sk w ith a horizontal 
line drawoi between them. The length of the horizontal line represents the time it lakes to complete the task. The 
chart comprises all the tasks needed to complete a project spread out over a time line to illustrate how lire long 
the project may take to complete. Independent parallel tusks can be shown above or below «nher tasks, and tasks 
dependent upon another task being finished first follow aficf the first task. The chart was designed to show the 
dependencies and the cntical points in the project that need lo he managed closely 

GIF - An acronym for Graphic Interchange Format. A phoiiigraphic file tormai tlevelo(KHl b\ C'ompuScrvei, u 
Internet Scrv'ice Provider like American On-Line, to allow ihcir users to exchange phiilos A common graphic 
format used on the Internet. Compare with JPEG and SGML. 

HTML - An acronym for HyperText Markup Language. Hl.Ml. is an Internet standard used to exchange 
document information semantically, leaving the client application, called a browser, to determine the proper way 
to display a document 

Informix - A large-scale commercial database company, w ith a database application by the same name. Compare 
with Ingress. Microsoft Access, Microsoft SQL Server. Oracle, and Sybase. 

Ingress - A medium scale commercial database application available from a company called Computer 
Associates. Compare with Informix, Microsoft Access. Microsoft SQI. .Server, Oracle, and Syba.se. 

Intelligence Community - The agencies and services of a government that conduct intelligence activities. In tlie 
United States the Intelligence Community is defined by membership on the National F-oreign Intelligence Board. 
Currently there arc fourteen agencies and departments having membership on this board They arc tlie Federal 
Bureau of Investigation, the Central Intelligence Agency, the Army, the Navy, the Air Force, the Marines, the 
U.S. Coast Guard, the Department of Treasur)', the Depanmeni of Energy, the Department of State, the National 
Security Agency, the Defense Intelligence Agency, the National Image and Mapping Agency, and the National 
Reconnaissance Office. 

JPEG - An acronym for Jet Propulsion Exchange Graphic. .A photographic file format developed by the Jet 
Propulsion Labs to allow photos from space to be shared with scientists over the Internet. A very common 
standard used on the Internet. Compare with GIF and SGMI.. 

Microsoft Access - A commercial software database fnim Microsoft Corporation that typically wi^rks only for 
one user or small business applications. Compare with Inlormix. Ingress. Microsoft SQI. Server, Oracle, and 
Sybase. 

Microsoft SQL Server - A commeicial software daiaba,sc from Microsoft Corporation that is designed for larger 
scale use than Acces.s, Compare with Informix, Ingress. Micricsoft Acce.ss. Oracle, and Sybase 

ODBC - An acronym for Open DataBase Connectivity. An industry standard u.sed to allow third party software 
to connect to any ODBC-compliant database. Compare with XML. 
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Oracle — A conlI^«^rcial database conipan). ^Mth a database application by the same name designed for large, 
medium and sittall-scale usage. Compare v^ith Iniormiv. Ingress, Microsoft Access, Microsoft SQI.. Server, and 
Sybase. 

Ot'der oj Hattie - The defined steps a coiiniry or eniiiy gnies through to ready themselves for offensive attack or 
defensive position. Compare with AsNinnuMiic Order of Battle and Signature. 

PDF - A file extensmn. Files ending with “.piif ate written in Adolie Acrobat format. Adofie offers a iree file 
viewer called Acrobat ti> allow users to read and print tliese files. 

Plotter ^ A printing device used to print ver\ large giaphics. like posters, blue prints, charts, etc. D-Size plotters 
handle paper about three or tour feet w ide 

Phtft’in — Softvvare that d^>e.s not w'ork alone as an application, but plugs into a third-party software application, 
and provide.^ that third-party applicaiii»n with enhanced capabilities it natively does not have. Sec API. 

SGML - An acronym fi»r Standardized ciruphic Markup Language. SGML is a standard from which other mark¬ 
up languages evolved. Fiir instance. H I ML is a sub-standard of SGML. SGML is used lo de.scribe graphics in a 
standardiz.e way so they can he freely slurred Compare with JPFIG and GIF. 

Sigtujture - Tlie unique characteristics that make up an intelligence event. For instance, if a person buys a ski 
mask, a handgun, paper, pen. and .steaks a car is probably preparing to rob a bonk The steps used to prepare for 
the robbery is called an order of battle, and the unique steps u.sed to identify it’s a bank robbery is called the 
signature. Compare w ith Order of Battle and Asymmetric Order of Battle. 

Sybase — A commercial database company, w ith a large-scale database application by the same name. Compare 
with Informix. Ingress. Microsoft Access, Micro.Nolt SQL Server, and Oracle. 

Visio - A Micro.sofi .software product name, 'fhe application is designed to draw schematics, cliarts and diagrams 

Wizard-iike Interface - A common interface u.sed to install applications on Micro.soft Windows. Tlic interface 
asks the user a series of questions. Based on the answers given, the wizard automates the process to assist the 
user. 

XML - Acronym for extensible Markup Language. An Internet standard used lo allow anyone understanding 
XML to interpret data in any source. Compare w ith ODBC. 
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Good online tutorial walks user 
through 7 activities with 
explanations and rationales. 

Language a bit dense 

sometimes. Y PDF & HTML formats 
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NAME 

ORGANIZATION 

POSITION 

PHONE 

EMAIL 

Greg Jannarone 

NAIC/BPB 

Primary Customer 

937-656-1055 

auQust.iannaronetS^wDafb.af.mil 

Eric Braeden 

NAIC/BPB 

NASIC Lead 

937-910-6490 

BraedenO^sbcalobal.net 

Ken Hammerle 

NAIC/BPB 

NASIC Analyst 

937-656-1055 

kenneth.hammerle(®WDafb.af.mil 

Gil Kuperman 

AFRL/HECA 

Contract Tech Lead 

937-255-3727 

aiibert.kuDermanO>wDafb.af.mil 

Capt Tim Gameros 

AFRL/HECA 

PSYOP Researcher 

937-255-4046 

timothv.aameros(®wDafb.af.mil 

Don Monk 

AFRL/HECA 

COAST (IWPC) 


donald.monk(S>wDafb. af.mil 

Dave Morion 

AFRL 

M&S Lead 

937-255-9349 

david.mortonO)wDafb.af.mil 

Dr. Jon Pfautz 

Charles River Analytics 

OCCAM 

617-491-3474 x541 

iofautzO^cra.com 

Karen Harper 

Charles River Analytics 

Sample. Grade 

617-491-3474 x533 

kharoerO^cra.com 

Dr. Mark Sheehan 

AFRL/HE 

CRA SBIR PM 

937-255-8806 


Bill Curtice 

ASC/ENM 

M&S Division Chief 

937-904-4409 

wiiliam.curticeO^WDafb.af.mii 

Joe Reiman 

ASC/ENM 

M&S Division 

937-904-4553 


Larry Daniel 

SRA 

Project Manager 

937-910-6433 

iarrv daniei(3^sra.com 

Mike Zywien 

SRA 

Tech Lead 

937-910-6425 

michael zvwienOsra.com 

Elisabeth Fitzhugh 

SRA 

Human Factors 

937-910-411 

elisabeth fitzhuah^sra.com 

Aaron Bryant 

SRA 

intelligence 

937-910-6426 

aaron brvantO>sra.com 

Ratna Bearavolu 

SRA 

Software 

937-910-6455 

ratna bearavoiuOlsra.com 

Rick Raftery 

SRA 

HUMINT 

703-684-2347 

rick rafterv®sra.com 

Travis Lynch 

JWAC 

JEMNA 

540-653-3757 


Phil Desmans 

JWAC 

JEMNA 

540-653-5757 


Bob Molepske 

JWAC - Raytheon 

JEMNA - Toffler 

703-413-5337 


Ken Moak 

AT&T 

BA&PS 

937-320-4571 

kmoakO^att.com 

Lt Col Bob Rushing 

JIOC 

BA&PS 

210-977-2484 


William Swart 

JIOC 



william.swartO^iioc.osis.Qov 

Mike Millen 

JIOC 


210-977-2483 


Larry Willis 

DARPA 


(703) 696-7448 

IwiilisO^darDa.mil 

Joe Schaff 

NAVAIR 


(301) 904-3759 

SchaffJBtS^na vair.navv.mil 

Vincente Garcia 

New Mexico State Liniv. 



vcaarcia^DSI.nmsu.edu 

Ray Buettner 

Naval Postgraduate School 


(831) 656-3387 

rrbuettntanDS.navY.mil 

Dr. Julie Rosen 

SAIC 

SIAM 

703-676-7354 

iuiie.a.rosentasaic.com 

Gary Fauss 

MRC 

PRFECT 

505-301-2591 

aarv.fausstakirtland.af.mil 

Col Jack Summe 





Mrs. Graham (Sec) 

USA 4th POG - Ft Bragg 

Group CC 

910-432-6392 


Col Alan Snyder 

AIA/DO-2 

AF PSYOP PMO IPT 

210-977-2116 


Frank Goldstein 

DO-2 


210-579-3940 


Glenn Kinder 

DO-2 

XP Chief of PSYOP 

210-977-2116 

alenn.kindertaiackland.af.mil 

Lt Col Doug Jaquish 

ASC/RAB 

Commando Solo PM 

255-2211 

douQ.iaauishtawDafb.af.mii 

Ron Swartzwelder 

NSA 

ASC Rep @ RA 

255-4326 

ronald.swartzweldertawDafb.af.mil 

Cindy Farkus 

NSA 


301-688-7111 


Bert Head 

NSA 

Dep Ops Rsrch M&S 

301-688-2857 


Bill Ryder 

NSA 

Tech Lead Ops Rsrch M&S 

301-688-4648 

whrvdertansa.QOv 

Lt Col Francine Goode 

NSA 

Ops Rsrch M&S 

301-688-2869 


Dr Tom Smith 

NSA 

DPLF Tool in OR M&S 

301-688-2851 


Jeanne Jones 

DIA 

AFMC rep (@ NAIC) 

257-3725 


Col Jeff Buckmelter 

SOCOM 

Mr. IWfor SOCOM 



Tim Menke 

SIMAF 

ASC M&S 

937-255-1276 


Randy Levine 

SIMAF 

ASC M&S 

937-255-0672 


Brooke McNally 

SIMAF 

ASC M&S 

937-255-0625 


Nicolas Rummelt 

Eglin AFB, FL 

Ex ASC M&S Expert 

850-882-3910x2320 
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Project Glossary 


Accessibility - Measure of whether a given target of interest can be reached 
(physically or by other means) to exploit a vulnerability 

Analytical Hierarchy Process - A multi-criteria decision analysis technique that 
uses a prioritized hierarchy of criteria to aid a decision process 

Alterables - Aspects of a given problem that can be manipulated, traded, or 
changed to help drive a solution 

Bayes’ Theorem - The posterior probability distribution for a given set of 
random variables can be calculated given an exact value for other variables 

Bayesian/Belief/Causal/Probabiiity Nets - Networks (nodes and links) used to 
represent dependencies between random variables; nodes (parent and child) are 
associated with conditional probability tables, links represent dependencies 
between nodes 

Commercial Off-The-Shelf (COTS) - Technologies, products, and services that 
can be purchased directly from commercial vendors 

Courses of Action (COA) - Alternative ways to approach and address a given 
problem 

Criteria Model - A hierarchal structure with a top level goal, criteria and sub¬ 
criteria that affect the goal, and a bottom level set of alternatives; each element 
of the model is assigned weights, and the weights determine the influence one 
element of the model has on another 

Data Mining - Methods and technologies associated with finding a desired piece 
of information in accessible data (text, video, audio, etc.) files 

Decision Node Support - Support for nodes that can be used to represent the 
decisions and the alternatives that the user is faced with. 

Decision Support System (DSS) - Computer-aided tools that emulate the 
reasoning process of a human expert making decisions 

Decision Trees - Chronological model of a problem depicted with roots and 
branches; roots are decision nodes, branches represent various possibilities of 
the decision node (branches lead to decision nodes, chance nodes, or result 
nodes) 

Decision/Utility Theory - A problem solving approach that strives to maximize a 
given utility function 
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Government Off-The-Shelf (GOTS) - Technologies, products, and sen/ices that 
can be acquired directly from other government agencies 

Inference Algorithm - Algorithms that calculate the posterior probability of the 
variables (nodes) in the Bayesian net, given exact values of some variables 

Influence Diagrams - Consists of a Bayesian Net, a Decision Node (alternative 
course of action), and a Value Node (value or utility of a given outcome): returns 
the action with the highest utility 

Influence Operations - Operations focused on affecting the perceptions and 
behaviors of people, leaders, groups, or entire populations. The means can be 
physical or informational. 

Mixed Graph Support - Support for both directed and undirected links in a 
model 

Modeling - Process of representing a given problem or reality with assigned 
values, weights, beliefs, etc. and using inference techniques to answer queries 

Psychological Effects-Based Operations (PEBO) - the deliberate use of the 
USAF core operational competencies and its enabling technologies as a 
psychological instrument. 

Requirements Trace - A structured methodology to tie products, services, and 
technologies and components of products, services, and technologies to 
technical and operational requirements 

Rules of Engagement (ROE) - a set of firm rules, usually framed by national 
policy and implemented by the Operational Commander, that govern the conduct 
of operations in a given theater 

Social Network Analysis - Social network analysis [SNA] is the mapping and 
measuring of relationships and flows between people, groups, organizations, 
computers or other information/knowledge processing entities. The nodes in the 
network are the people and groups while the links show relationships or flows 
between the nodes. SNA provides both a visual and a mathematical analysis of 
human relationships. (Definition from www.orgnet.com) 

Structure Learning - Technique that automatically generates a Bayesian Net, 
given data (in table format) 

Susceptibility - A predictable path to exploit a given vulnerability 
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Text Mining - Methods and technologies associated with finding a desired piece 
of information in accessible text files 

Virtual Evidence Support - Support to enter evidence into the Bayesian net 
which the user is uncertain about. 

Vulnerability - Characteristics of a chosen target that planners and analysts can 
leverage and exploit 
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